<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta name="generator" content="hevea 2.09">
<style type="text/css">
.c000{border-spacing:6px;border-collapse:separate;}
.c001{color:#007F00}
.c002{color:blue}
.c003{color:red}
.c004{font-family:monospace}
.c005{font-size:small}
.c006{font-size:xx-large}
.c007{font-style:italic}
.c008{font-style:italic;color:red}
.c009{font-style:italic;font-weight:bold}
.c010{font-weight:bold}
.c011{padding:1ex;background-color:#E5EDFF;}
.c012{text-align:center}
.c013{text-align:center;white-space:nowrap}
.c014{text-align:left;white-space:nowrap}
.c015{text-align:right;white-space:nowrap}
.c016{vertical-align:middle}
.li-itemize{margin:1ex 0ex;}
.toc{list-style:none;}
div.footnotetext P{margin:0px; text-indent:1em;}
.title{margin:2ex auto;text-align:center}
.titlemain{margin:1ex 2ex 2ex 1ex;}
.titlerest{margin:0ex 2ex;}
.flushleft{text-align:left;margin-left:0ex;margin-right:auto;}
.flushright{text-align:right;margin-left:auto;margin-right:0ex;}
div table{margin-left:inherit;margin-right:inherit;margin-bottom:2px;margin-top:2px}
td table{margin:auto;}
table{border-collapse:collapse;}
td{padding:0;}
pre{text-align:left;margin-left:0ex;margin-right:auto;}
blockquote{margin-left:4ex;margin-right:4ex;text-align:left;}
td p{margin:0px;}
.hbar{border:none;height:2px;width:100%;background-color:black;}
.display{border-collapse:separate;border-spacing:2px;width:auto; border:none;}
.dcell{white-space:nowrap;padding:0px; border:none;}
.dcenter{margin:0ex auto;}
.mouselstlisting{font-family:monospace;margin-right:auto;margin-left:0pt;text-align:left}
.mouselstlisting{font-family:monospace;margin-right:auto;margin-left:0pt;text-align:left}
</style>

<META name="Author" content="Julien Mairal">
<link rel="stylesheet" href="doc_spams.css">
<title>SPAMS: a SPArse Modeling Software, v2.5
</title>
</head>
<body>
<!--HEVEA command line is: /usr/bin/hevea -fix -O article.hva macrocss.hva doc_spams.tex -->
<!--CUT STYLE article--><!--CUT DEF section 1 --><table class="title"><tr><td style="padding:1ex"><h1 class="titlemain">SPAMS: a SPArse Modeling Software, v2.5</h1><h3 class="titlerest">Julien Mairal <br>
 <span class="c004">julien.mairal@m4x.org</span> <br>
  <br>
 Python and R interfaces by Jean-Paul Chieze
 <br>
 Archetypal Analysis by Yuansi Chen
 <br>
 and the output of numerous collaborations with <br>
 Rodolphe Jenatton, Francis Bach, Jean Ponce, <br>
Guillaume Obozinski, Bin Yu, Guillermo Sapiro, <br>
and Zaid
Harchaoui. </h3></td></tr>
</table><!--TOC section id="sec1" Contents-->
<h2 class="section" id="sec1">Contents</h2><!--SEC END --><ul class="toc"><li class="li-toc">
<a href="#sec2">1  Introduction</a>
</li><li class="li-toc"><a href="#sec3">2  Installation</a>
</li><li class="li-toc"><a href="#sec4">3  Dictionary Learning and Matrix Factorization Toolbox</a>
<ul class="toc"><li class="li-toc">
<a href="#sec5">3.1  Function mexTrainDL</a>
</li><li class="li-toc"><a href="#sec6">3.2  Function mexTrainDL_Memory</a>
</li><li class="li-toc"><a href="#sec7">3.3  Function mexStructTrainDL</a>
</li><li class="li-toc"><a href="#sec8">3.4  Function nmf</a>
</li><li class="li-toc"><a href="#sec9">3.5  Function nnsc</a>
</li><li class="li-toc"><a href="#sec10">3.6  Function mexArchetypalAnalysis</a>
</li></ul>
</li><li class="li-toc"><a href="#sec11">4  Sparse Decomposition Toolbox</a>
<ul class="toc"><li class="li-toc">
<a href="#sec12">4.1  Function mexOMP</a>
</li><li class="li-toc"><a href="#sec13">4.2  Function mexOMPMask</a>
</li><li class="li-toc"><a href="#sec14">4.3  Function mexRidgeRegression</a>
</li><li class="li-toc"><a href="#sec15">4.4  Function mexLasso</a>
</li><li class="li-toc"><a href="#sec16">4.5  Function mexLassoWeighted</a>
</li><li class="li-toc"><a href="#sec17">4.6  Function mexLassoMask</a>
</li><li class="li-toc"><a href="#sec18">4.7  Function mexCD</a>
</li><li class="li-toc"><a href="#sec19">4.8  Function mexSOMP</a>
</li><li class="li-toc"><a href="#sec20">4.9  Function mexL1L2BCD</a>
</li><li class="li-toc"><a href="#sec21">4.10  Function mexSparseProject</a>
</li><li class="li-toc"><a href="#sec22">4.11  Function mexDecompSimplex</a>
</li></ul>
</li><li class="li-toc"><a href="#sec23">5  Proximal Toolbox</a>
<ul class="toc"><li class="li-toc">
<a href="#sec24">5.1  Regularization Functions</a>
</li><li class="li-toc"><a href="#sec25">5.2  Function mexProximalFlat</a>
</li><li class="li-toc"><a href="#sec26">5.3  Function mexProximalTree</a>
</li><li class="li-toc"><a href="#sec27">5.4  Function mexProximalGraph</a>
</li><li class="li-toc"><a href="#sec28">5.5  Function mexProximalPathCoding</a>
</li><li class="li-toc"><a href="#sec29">5.6  Function mexEvalPathCoding</a>
</li><li class="li-toc"><a href="#sec30">5.7  Problems Addressed</a>
<ul class="toc"><li class="li-toc">
<a href="#sec31">5.7.1  Regression Problems with the Square Loss</a>
</li><li class="li-toc"><a href="#sec32">5.7.2  Classification Problems with the Logistic Loss</a>
</li><li class="li-toc"><a href="#sec33">5.7.3  Multi-class Classification Problems with the Softmax Loss</a>
</li><li class="li-toc"><a href="#sec34">5.7.4  Multi-task Regression Problems with the Square Loss</a>
</li><li class="li-toc"><a href="#sec35">5.7.5  Multi-task Classification Problems with the Logistic Loss</a>
</li><li class="li-toc"><a href="#sec36">5.7.6  Multi-task and Multi-class Classification Problems with the Softmax Loss</a>
</li></ul>
</li><li class="li-toc"><a href="#sec37">5.8  Function mexFistaFlat</a>
</li><li class="li-toc"><a href="#sec38">5.9  Function mexFistaTree</a>
</li><li class="li-toc"><a href="#sec39">5.10  Function mexFistaGraph</a>
</li><li class="li-toc"><a href="#sec40">5.11  Function mexFistaPathCoding</a>
</li><li class="li-toc"><a href="#sec41">5.12  Function solverPoisson</a>
</li><li class="li-toc"><a href="#sec42">5.13  Function mexIncrementalProx</a>
</li><li class="li-toc"><a href="#sec43">5.14  Function mexStochasticProx</a>
</li></ul>
</li><li class="li-toc"><a href="#sec44">6  A few Functions for manipulating images</a>
<ul class="toc"><li class="li-toc">
<a href="#sec45">6.1  Function mexExtractPatches</a>
</li><li class="li-toc"><a href="#sec46">6.2  Function mexCombinePatches</a>
</li><li class="li-toc"><a href="#sec47">6.3  Function mexConvFistaFlat</a>
</li></ul>
</li><li class="li-toc"><a href="#sec48">7  Miscellaneous Functions</a>
<ul class="toc"><li class="li-toc">
<a href="#sec49">7.1  Function mexConjGrad</a>
</li><li class="li-toc"><a href="#sec50">7.2  Function mexBayer</a>
</li><li class="li-toc"><a href="#sec51">7.3  Function mexCalcAAt</a>
</li><li class="li-toc"><a href="#sec52">7.4  Function mexCalcXAt</a>
</li><li class="li-toc"><a href="#sec53">7.5  Function mexCalcXY</a>
</li><li class="li-toc"><a href="#sec54">7.6  Function mexCalcXYt</a>
</li><li class="li-toc"><a href="#sec55">7.7  Function mexCalcXtY</a>
</li><li class="li-toc"><a href="#sec56">7.8  Function mexInvSym</a>
</li><li class="li-toc"><a href="#sec57">7.9  Function mexNormalize</a>
</li><li class="li-toc"><a href="#sec58">7.10  Function mexSort</a>
</li><li class="li-toc"><a href="#sec59">7.11  Function mexDisplayPatches</a>
</li><li class="li-toc"><a href="#sec60">7.12  Function mexCountPathsDAG</a>
</li><li class="li-toc"><a href="#sec61">7.13  Function mexRemoveCyclesGraph</a>
</li><li class="li-toc"><a href="#sec62">7.14  Function mexCountConnexComponents</a>
</li><li class="li-toc"><a href="#sec63">7.15  Function mexGraphOfGroupStruct</a>
</li><li class="li-toc"><a href="#sec64">7.16  Function mexGroupStructOfString</a>
</li><li class="li-toc"><a href="#sec65">7.17  Function mexReadGroupStruct</a>
</li><li class="li-toc"><a href="#sec66">7.18  Function mexTreeOfGroupStruct</a>
</li><li class="li-toc"><a href="#sec67">7.19  Function mexSimpleGroupTree</a>
</li></ul>
</li><li class="li-toc"><a href="#sec68">A  Duality Gaps with Fenchel Duality</a>
<ul class="toc">
<ul class="toc"><li class="li-toc">
<a href="#sec69">A.0.1  Duality Gaps without Intercepts</a>
</li><li class="li-toc"><a href="#sec70">A.0.2  Duality Gaps with Intercepts</a>
</li></ul>
</ul>
</li></ul>
<!--TOC section id="sec2" Introduction-->
<h2 class="section" id="sec2">1  Introduction</h2><!--SEC END --><p>
SPAMS (SPArse Modeling Software) is an open-source optimization toolbox for
sparse estimation with licence GPLv3. It implements algorithms for solving
machine learning and signal processing problems involving sparse
regularizations.</p><p>The library is coded in C++, is compatible with Linux, Mac, and Windows 32bits
and 64bits Operating Systems. It is interfaced with Matlab, R and Python, but
can be called from any C++ application (by hacking the code a bit).</p><p>It requires an implementation of BLAS and LAPACK for performing linear algebra
operations. The ones shipped with matlab and R can be used, but also external
libraries such as atlas, the netlib implementation, or the Intel Math Kernel
Library can be used. It also exploits multi-core CPUs when this feature is
supported by the compiler, through OpenMP.</p><p>The current licence is GPLv3, which is available at
<span class="c004">http://www.gnu.org/licenses/gpl.html</span>. For other licensing possibilities 
allowing its use in proprietary softwares, please contact the author.</p><p>Version 2.5 of SPAMS is divided into several “toolboxes” and has a few
additional miscellaneous functions:
</p><ul class="itemize"><li class="li-itemize">
The <span class="c010">Dictionary learning and matrix factorization toolbox</span>
contains the online learning technique of [<a href="#mairal7">20</a>, <a href="#mairal9">21</a>] and its
variants for solving various matrix factorization problems:
<ul class="itemize"><li class="li-itemize">
dictionary Learning for sparse coding;
</li><li class="li-itemize">sparse principal component analysis (seen as a sparse matrix factorization problem);
</li><li class="li-itemize">non-negative matrix factorization;
</li><li class="li-itemize">non-negative sparse coding;
</li><li class="li-itemize">dictionary learning with structured sparsity;
</li><li class="li-itemize">archetypal analysis [<a href="#Cut94">7</a>, <a href="#ChenCVPR">37</a>].
</li></ul>
</li><li class="li-itemize">The <span class="c010">Sparse decomposition toolbox</span> contains efficient implementations of
<ul class="itemize"><li class="li-itemize">
Orthogonal Matching Pursuit, (or Forward Selection) [<a href="#weisberg">35</a>, <a href="#mallat4">27</a>];
</li><li class="li-itemize">the LARS/homotopy algorithm [<a href="#osborne">30</a>, <a href="#efron">9</a>] (variants for solving Lasso and Elastic-Net problems);
</li><li class="li-itemize">a weighted version of LARS; 
</li><li class="li-itemize">OMP and LARS when data comes with a binary mask;
</li><li class="li-itemize">a coordinate-descent algorithm for ℓ<sub>1</sub>-decomposition problems [<a href="#fu">12</a>, <a href="#friedman">10</a>, <a href="#wu">36</a>]; 
</li><li class="li-itemize">a greedy solver for simultaneous signal approximation as defined in [<a href="#tropp2">34</a>, <a href="#tropp3">33</a>] (SOMP);
</li><li class="li-itemize">a solver for simulatneous signal approximation with ℓ<sub>1</sub>/ℓ<sub>2</sub>-regularization based on block-coordinate descent;
</li><li class="li-itemize">a homotopy method for the Fused-Lasso Signal Approximation as defined in [<a href="#friedman">10</a>] with the homotopy method presented in the appendix of [<a href="#mairal9">21</a>];
</li><li class="li-itemize">a tool for projecting efficiently onto a few convex sets
inducing sparsity such as the ℓ<sub>1</sub>-ball using the method of
[<a href="#brucker">3</a>, <a href="#maculan">18</a>, <a href="#duchi">8</a>], and Elastic-Net or Fused Lasso constraint sets as
proposed in the appendix of [<a href="#mairal9">21</a>].
</li><li class="li-itemize">an active-set algorithm for simplex decomposition problems [<a href="#ChenCVPR">37</a>].
</li></ul>
</li><li class="li-itemize">The <span class="c010">Proximal toolbox</span>: An implementation of proximal methods
(ISTA and FISTA [<a href="#beck">1</a>]) for solving a large class of sparse approximation
problems with different combinations of loss and regularizations. One of the main
features of this toolbox is to provide a robust stopping criterion based on
<em>duality gaps</em> to control the quality of the optimization, whenever
possible. It also handles sparse feature matrices for large-scale problems. The following regularizations are implemented:
<ul class="itemize"><li class="li-itemize">
Tikhonov regularization (squared ℓ<sub>2</sub>-norm);
</li><li class="li-itemize">ℓ<sub>1</sub>-norm, ℓ<sub>2</sub>, ℓ<sub>∞</sub>-norms;
</li><li class="li-itemize">Elastic-Net [<a href="#zou">39</a>];
</li><li class="li-itemize">Fused Lasso [<a href="#tibshirani2">32</a>];
</li><li class="li-itemize">tree-structured sum of ℓ<sub>2</sub>-norms (see [<a href="#jenatton3">15</a>, <a href="#jenatton4">16</a>]);
</li><li class="li-itemize">tree-structured sum of ℓ<sub>∞</sub>-norms (see [<a href="#jenatton3">15</a>, <a href="#jenatton4">16</a>]);
</li><li class="li-itemize">general sum of ℓ<sub>∞</sub>-norms (see [<a href="#mairal10">22</a>, <a href="#mairal13">23</a>]);
</li><li class="li-itemize">mixed ℓ<sub>1</sub>/ℓ<sub>2</sub>-norms on matrices [<a href="#yuan">38</a>, <a href="#obozinski">29</a>];
</li><li class="li-itemize">mixed ℓ<sub>1</sub>/ℓ<sub>∞</sub>-norms on matrices [<a href="#yuan">38</a>, <a href="#obozinski">29</a>];
</li><li class="li-itemize">mixed ℓ<sub>1</sub>/ℓ<sub>2</sub>-norms on matrices plus ℓ<sub>1</sub> [<a href="#sprechmann">31</a>, <a href="#Friedman2010">11</a>];
</li><li class="li-itemize">mixed ℓ<sub>1</sub>/ℓ<sub>∞</sub>-norms on matrices plus ℓ<sub>1</sub>;
</li><li class="li-itemize">group-lasso with ℓ<sub>2</sub> or ℓ<sub>∞</sub>-norms;
</li><li class="li-itemize">group-lasso+ℓ<sub>1</sub>;
</li><li class="li-itemize">multi-task tree-structured sum of ℓ<sub>∞</sub>-norms (see [<a href="#mairal10">22</a>, <a href="#mairal13">23</a>]);
</li><li class="li-itemize">trace norm;
</li><li class="li-itemize">ℓ<sub>0</sub> pseudo-norm (only with ISTA);
</li><li class="li-itemize">tree-structured ℓ<sub>0</sub> (only with ISTA);
</li><li class="li-itemize">rank regularization for matrices (only with ISTA);
</li><li class="li-itemize">the path-coding penalties of [<a href="#mairal14">24</a>].
</li></ul>
All of these regularization functions can be used with the following losses
<ul class="itemize"><li class="li-itemize">
square loss;
</li><li class="li-itemize">square loss with missing observations; 
</li><li class="li-itemize">logistic loss, weighted logistic loss;
</li><li class="li-itemize">multi-class logistic.
</li></ul>
This toolbox can also enforce non-negativity constraints, handle intercepts and
sparse matrices. There are also a few additional undocumented functionalities,
which are available in the source code.
For some combinations of loss and regularizers, stochastic and incremental proximal
gradient solvers are also implemented [<a href="#mairal15">26</a>, <a href="#mairal16">25</a>].
</li><li class="li-itemize">A few tools for performing linear algebra operations such as a
conjugate gradient algorithm, manipulating sparse matrices and graphs.
</li></ul><p>The toolbox was written by Julien Mairal at INRIA, with the collaboration of
Francis Bach (INRIA), Jean Ponce (Ecole Normale Supérieure), Guillermo Sapiro
(University of Minnesota), Guillaume Obozinski (INRIA) and Rodolphe Jenatton
(INRIA).</p><p>R and Python interfaces have been written by Jean-Paul Chieze (INRIA).
The archetypal analysis implementation was written by Yuansi Chen, during
an internship at INRIA, with the collaboration of Zaid Harchaoui.</p>
<!--TOC section id="sec3" Installation-->
<h2 class="section" id="sec3">2  Installation</h2><!--SEC END --><p>
The toolbox used to come with pre-compiled binaries for various plateforms. 
Now the sources are available and you will have to compile it yourself.</p><p>The user has the choice of the BLAS library, but the Intel MKL is recommended
for the best performance. Note that the builtin blas library of Matlab is a
version of the Intel MKL (not the most recent one though), and offers 
a good performance.</p><p>The folder <code>doc</code> contains the documentation in pdf and html. <code>build/</code>
contains the binary files, including the help for each command.
<code>test_release</code> contains various matlab scripts which call the different
functions of this toolbox.</p><p>The software package comes also with a script bash that has to be used to
launch matlab for Linux and/or Mac OS versions. The install procedure is
described in a file called <code>HOW_TO_INSTALL</code> and in the file
<code>compile.m</code> for the Matlab version.</p>
<!--TOC section id="sec4" Dictionary Learning and Matrix Factorization Toolbox-->
<h2 class="section" id="sec4">3  Dictionary Learning and Matrix Factorization Toolbox</h2><!--SEC END --><p>
This is the section for dictionary learning and matrix factorization, corresponding to [<a href="#mairal7">20</a>, <a href="#mairal9">21</a>].</p>
<!--TOC subsection id="sec5" Function mexTrainDL-->
<h3 class="subsection" id="sec5">3.1  Function mexTrainDL</h3><!--SEC END --><p>
This is the main function of the toolbox, implementing the learning algorithms of [<a href="#mairal9">21</a>]. 
Given a training set <span class="c009">x</span><sup>1</sup>,…, . It aims at solving
</p><table class="display dcenter"><tr class="c016"><td class="dcell">
</td><td class="dcell"><table class="display"><tr><td class="dcell c012">&nbsp;</td></tr>
<tr><td class="dcell c012">min</td></tr>
<tr><td class="dcell c012"><span class="c009">D</span> ∈ <span class="c008">C</span></td></tr>
</table></td><td class="dcell"> </td><td class="dcell"><table class="display"><tr><td class="dcell c012">&nbsp;</td></tr>
<tr><td class="dcell c012">lim</td></tr>
<tr><td class="dcell c012"><span class="c007">n</span> → +∞</td></tr>
</table></td><td class="dcell"> </td><td class="dcell"><table class="display"><tr><td class="dcell c012">1</td></tr>
<tr><td class="hbar"></td></tr>
<tr><td class="dcell c012"><span class="c007">n</span></td></tr>
</table></td><td class="dcell"> </td><td class="dcell"><table class="display"><tr><td class="dcell c012"><span class="c007">n</span></td></tr>
<tr><td class="dcell c012"><span class="c006">∑</span></td></tr>
<tr><td class="dcell c012"><span class="c007">i</span>=1</td></tr>
</table></td><td class="dcell"> </td><td class="dcell"><table class="display"><tr><td class="dcell c012">&nbsp;</td></tr>
<tr><td class="dcell c012">min</td></tr>
<tr><td class="dcell c012">α<sup><span class="c007">i</span></sup></td></tr>
</table></td><td class="dcell"> </td><td class="dcell">⎛<br>
⎜<br>
⎝</td><td class="dcell"><table class="display"><tr><td class="dcell c012">1</td></tr>
<tr><td class="hbar"></td></tr>
<tr><td class="dcell c012">2</td></tr>
</table></td><td class="dcell"> ||<span class="c009">x</span><sup><span class="c007">i</span></sup>−<span class="c009">D</span>α<sup><span class="c007">i</span></sup>||<sub>2</sub><sup>2</sup> + ψ(α<sup><span class="c007">i</span></sup>)</td><td class="dcell">⎞<br>
⎟<br>
⎠</td><td class="dcell">.
    (1)</td></tr>
</table><p>
ψ is a sparsity-inducing regularizer and <span class="c008">C</span> is a constraint set for the dictionary. As shown in [<a href="#mairal9">21</a>] 
and in the help file below, various combinations can be used for ψ and <span class="c008">C</span> for solving different matrix factorization problems.
What is more, positivity constraints can be added to α as well. The function admits several modes for choosing the optimization parameters, using the parameter-free strategy proposed in [<a href="#mairal7">20</a>], or using the parameters <span class="c007">t</span><sub>0</sub> and ρ presented
in [<a href="#mairal9">21</a>]. <span class="c010">Note that for problems of a reasonable size, and when </span>ψ<span class="c010"> is the </span>ℓ<sub>1</sub><span class="c010">-norm, 
the function mexTrainDL_Memory can be faster but uses more memory.</span> </p><table class="lstframe c011"><tr><td class="mouselstlisting"><span class="c001">% <br>
% Usage:   [D [model]]=mexTrainDL(X,param[,model]);<br>
%          model is optional<br>
%<br>
% Name: mexTrainDL<br>
%<br>
% Description: mexTrainDL is an efficient implementation of the<br>
%     dictionary learning technique presented in<br>
%<br>
%     "Online Learning for Matrix Factorization and Sparse Coding"<br>
%     by Julien Mairal, Francis Bach, Jean Ponce and Guillermo Sapiro<br>
%     arXiv:0908.0050<br>
%     <br>
%     "Online Dictionary Learning for Sparse Coding"      <br>
%     by Julien Mairal, Francis Bach, Jean Ponce and Guillermo Sapiro<br>
%     ICML 2009.<br>
%<br>
%     Note that if you use param.mode=1 or 2, if the training set has a<br>
%     reasonable size and you have enough memory on your computer, you <br>
%     should use mexTrainDL_Memory instead.<br>
% <br>
%<br>
%     It addresses the dictionary learning problems<br>
%        1) if param.mode=0<br>
%     min_{D in C} (1/n) sum_{i=1}^n (1/2)||x_i-Dalpha_i||_2^2  s.t. ...<br>
%                                                  ||alpha_i||_1 &lt;= lambda<br>
%        2) if param.mode=1<br>
%     min_{D in C} (1/n) sum_{i=1}^n  ||alpha_i||_1  s.t.  ...<br>
%                                           ||x_i-Dalpha_i||_2^2 &lt;= lambda<br>
%        3) if param.mode=2<br>
%     min_{D in C} (1/n) sum_{i=1}^n (1/2)||x_i-Dalpha_i||_2^2 + ... <br>
%                                  lambda||alpha_i||_1 + lambda_2||alpha_i||_2^2<br>
%        4) if param.mode=3, the sparse coding is done with OMP<br>
%     min_{D in C} (1/n) sum_{i=1}^n (1/2)||x_i-Dalpha_i||_2^2  s.t. ... <br>
%                                                  ||alpha_i||_0 &lt;= lambda<br>
%        5) if param.mode=4, the sparse coding is done with OMP<br>
%     min_{D in C} (1/n) sum_{i=1}^n  ||alpha_i||_0  s.t.  ...<br>
%                                           ||x_i-Dalpha_i||_2^2 &lt;= lambda<br>
%        6) if param.mode=5, the sparse coding is done with OMP<br>
%     min_{D in C} (1/n) sum_{i=1}^n 0.5||x_i-Dalpha_i||_2^2 +lambda||alpha_i||_0  <br>
%                                           <br>
%<br>
%%     C is a convex set verifying<br>
%        1) if param.modeD=0<br>
%           C={  D in Real^{m x p}  s.t.  forall j,  ||d_j||_2^2 &lt;= 1 }<br>
%        2) if param.modeD=1<br>
%           C={  D in Real^{m x p}  s.t.  forall j,  ||d_j||_2^2 + ... <br>
%                                                  gamma1||d_j||_1 &lt;= 1 }<br>
%        3) if param.modeD=2<br>
%           C={  D in Real^{m x p}  s.t.  forall j,  ||d_j||_2^2 + ... <br>
%                                  gamma1||d_j||_1 + gamma2 FL(d_j) &lt;= 1 }<br>
%        4) if param.modeD=3<br>
%           C={  D in Real^{m x p}  s.t.  forall j,  (1-gamma1)||d_j||_2^2 + ... <br>
%                                  gamma1||d_j||_1 &lt;= 1 }<br>
<br>
%<br>
%     Potentially, n can be very large with this algorithm.<br>
%<br>
% Inputs: X:  double m x n matrix   (input signals)<br>
%               m is the signal size<br>
%               n is the number of signals to decompose<br>
%         param: struct<br>
%            param.D: (optional) double m x p matrix   (dictionary)<br>
%              p is the number of elements in the dictionary<br>
%              When D is not provided, the dictionary is initialized <br>
%              with random elements from the training set.<br>
%           param.K (size of the dictionary, optional is param.D is provided)<br>
%           param.lambda  (parameter)<br>
%           param.lambda2  (optional, by default 0)<br>
%           param.iter (number of iterations).  If a negative number is <br>
%              provided it will perform the computation during the<br>
%              corresponding number of seconds. For instance param.iter=-5<br>
%              learns the dictionary during 5 seconds.<br>
%           param.mode (optional, see above, by default 2) <br>
%           param.posAlpha (optional, adds positivity constraints on the<br>
%             coefficients, false by default, not compatible with <br>
%             param.mode =3,4)<br>
%           param.modeD (optional, see above, by default 0)<br>
%           param.posD (optional, adds positivity constraints on the <br>
%             dictionary, false by default, not compatible with <br>
%             param.modeD=2)<br>
%           param.gamma1 (optional parameter for param.modeD &gt;= 1)<br>
%           param.gamma2 (optional parameter for param.modeD = 2)<br>
%           param.batchsize (optional, size of the minibatch, by default <br>
%              512)<br>
%           param.iter_updateD (optional, number of BCD iterations for the dictionary<br>
%              update step, by default 1)<br>
%           param.modeParam (optimization mode).<br>
%              1) if param.modeParam=0, the optimization uses the <br>
%                 parameter free strategy of the ICML paper<br>
%              2) if param.modeParam=1, the optimization uses the <br>
%                 parameters rho as in arXiv:0908.0050<br>
%              3) if param.modeParam=2, the optimization uses exponential <br>
%                 decay weights with updates of the form <br>
%                 A_{t} &lt;- rho A_{t-1} + alpha_t alpha_t^T<br>
%           param.rho (optional) tuning parameter (see paper arXiv:0908.0050)<br>
%           param.t0 (optional) tuning parameter (see paper arXiv:0908.0050)<br>
%           param.clean (optional, true by default. prunes <br>
%              automatically the dictionary from unused elements).<br>
%           param.verbose (optional, true by default, increase verbosity)<br>
%           param.numThreads (optional, number of threads for exploiting<br>
%              multi-core / multi-cpus. By default, it takes the value -1,<br>
%              which automatically selects all the available CPUs/cores).<br>
%<br>
% Output: <br>
%         param.D: double m x p matrix   (dictionary)<br>
%<br>
% Note: this function admits a few experimental usages, which have not<br>
%     been extensively tested:<br>
%         - single precision setting <br>
%<br>
% Author: Julien Mairal, 2009</span></td></tr>
</table><p>The following piece of code illustrates how to use this function.
</p><table class="lstframe c011"><tr><td class="mouselstlisting"><span class="c002">clear</span> <span class="c002">all</span>;<br>
<br>
I=double(imread(<span class="c003">'data/lena.png'</span>))/255;<br>
<span class="c001">% extract 8 x 8 patches</span><br>
X=im2col(I,[8 8],<span class="c003">'sliding'</span>);<br>
X=X-repmat(<span class="c002">mean</span>(X),[<span class="c002">size</span>(X,1) 1]);<br>
X=X ./ repmat(<span class="c002">sqrt</span>(<span class="c002">sum</span>(X.^2)),[<span class="c002">size</span>(X,1) 1]);<br>
<br>
param.K=256;  <span class="c001">% learns a dictionary with 100 elements</span><br>
param.lambda=0.15;<br>
param.numThreads=-1; <span class="c001">% number of threads</span><br>
param.batchsize=400;<br>
param.verbose=false;<br>
<br>
param.iter=1000;  <span class="c001">% let us see what happens after 1000 iterations.<br>
<br>
%%%%%%%%%% FIRST EXPERIMENT %%%%%%%%%%%</span><br>
<span class="c002">tic</span><br>
D = mexTrainDL(X,param);<br>
t=<span class="c002">toc</span>;<br>
<span class="c002">fprintf</span>(<span class="c003">'time of computation for Dictionary Learning: %f\n'</span>,t);<br>
<br>
<span class="c002">fprintf</span>(<span class="c003">'Evaluating cost function...\n'</span>);<br>
alpha=mexLasso(X,D,param);<br>
R=<span class="c002">mean</span>(0.5*<span class="c002">sum</span>((X-D*alpha).^2)+param.lambda*<span class="c002">sum</span>(<span class="c002">abs</span>(alpha)));<br>
ImD=displayPatches(D);<br>
<span class="c002">subplot</span>(1,3,1);<br>
<span class="c002">imagesc</span>(ImD); <span class="c002">colormap</span>(<span class="c003">'gray'</span>);<br>
<span class="c002">fprintf</span>(<span class="c003">'objective function: %f\n'</span>,R);<br>
<span class="c002">drawnow</span>;<br>
<br>
<span class="c002">fprintf</span>(<span class="c003">'*********** SECOND EXPERIMENT ***********\n'</span>);<br>
<span class="c001">%%%%%%%%%% SECOND EXPERIMENT %%%%%%%%%%%<br>
% Train on half of the training set, then retrain on the second part</span><br>
X1=X(:,1:<span class="c002">floor</span>(<span class="c002">size</span>(X,2)/2));<br>
X2=X(:,<span class="c002">floor</span>(<span class="c002">size</span>(X,2)/2):<span class="c002">end</span>);<br>
param.iter=500;<br>
<span class="c002">tic</span><br>
[D model] = mexTrainDL(X1,param);<br>
t=<span class="c002">toc</span>;<br>
<span class="c002">fprintf</span>(<span class="c003">'time of computation for Dictionary Learning: %f\n'</span>,t);<br>
<span class="c002">fprintf</span>(<span class="c003">'Evaluating cost function...\n'</span>);<br>
alpha=mexLasso(X,D,param);<br>
R=<span class="c002">mean</span>(0.5*<span class="c002">sum</span>((X-D*alpha).^2)+param.lambda*<span class="c002">sum</span>(<span class="c002">abs</span>(alpha)));<br>
<span class="c002">fprintf</span>(<span class="c003">'objective function: %f\n'</span>,R);<br>
<span class="c002">tic</span><br>
<span class="c001">% Then reuse the learned model to retrain a few iterations more.</span><br>
param2=param;<br>
param2.D=D;<br>
[D model] = mexTrainDL(X2,param2,model);<br>
<span class="c001">%[D] = mexTrainDL(X,param);</span><br>
t=<span class="c002">toc</span>;<br>
<span class="c002">fprintf</span>(<span class="c003">'time of computation for Dictionary Learning: %f\n'</span>,t);<br>
<span class="c002">fprintf</span>(<span class="c003">'Evaluating cost function...\n'</span>);<br>
alpha=mexLasso(X,D,param);<br>
R=<span class="c002">mean</span>(0.5*<span class="c002">sum</span>((X-D*alpha).^2)+param.lambda*<span class="c002">sum</span>(<span class="c002">abs</span>(alpha)));<br>
<span class="c002">fprintf</span>(<span class="c003">'objective function: %f\n'</span>,R);<br>
<br>
<span class="c001">% let us add sparsity to the dictionary itself</span><br>
<span class="c002">fprintf</span>(<span class="c003">'*********** THIRD EXPERIMENT ***********\n'</span>);<br>
param.modeParam=0;<br>
param.iter=1000;<br>
param.gamma1=0.3;<br>
param.modeD=1;<br>
<span class="c002">tic</span><br>
[D] = mexTrainDL(X,param);<br>
t=<span class="c002">toc</span>;<br>
<span class="c002">fprintf</span>(<span class="c003">'time of computation for Dictionary Learning: %f\n'</span>,t);<br>
<span class="c002">fprintf</span>(<span class="c003">'Evaluating cost function...\n'</span>);<br>
alpha=mexLasso(X,D,param);<br>
R=<span class="c002">mean</span>(0.5*<span class="c002">sum</span>((X-D*alpha).^2)+param.lambda*<span class="c002">sum</span>(<span class="c002">abs</span>(alpha)));<br>
<span class="c002">fprintf</span>(<span class="c003">'objective function: %f\n'</span>,R);<br>
<span class="c002">tic<br>
subplot</span>(1,3,2);<br>
ImD=displayPatches(D);<br>
<span class="c002">imagesc</span>(ImD); <span class="c002">colormap</span>(<span class="c003">'gray'</span>);<br>
<span class="c002">drawnow</span>;<br>
<br>
<span class="c002">fprintf</span>(<span class="c003">'*********** FOURTH EXPERIMENT ***********\n'</span>);<br>
param.modeParam=0;<br>
param.iter=1000;<br>
param.gamma1=0.3;<br>
param.modeD=3;<br>
<span class="c002">tic</span><br>
[D] = mexTrainDL(X,param);<br>
t=<span class="c002">toc</span>;<br>
<span class="c002">fprintf</span>(<span class="c003">'time of computation for Dictionary Learning: %f\n'</span>,t);<br>
<span class="c002">fprintf</span>(<span class="c003">'Evaluating cost function...\n'</span>);<br>
alpha=mexLasso(X,D,param);<br>
R=<span class="c002">mean</span>(0.5*<span class="c002">sum</span>((X-D*alpha).^2)+param.lambda*<span class="c002">sum</span>(<span class="c002">abs</span>(alpha)));<br>
<span class="c002">fprintf</span>(<span class="c003">'objective function: %f\n'</span>,R);<br>
<span class="c002">tic<br>
subplot</span>(1,3,3);<br>
ImD=displayPatches(D);<br>
<span class="c002">imagesc</span>(ImD); <span class="c002">colormap</span>(<span class="c003">'gray'</span>);<br>
<span class="c002">drawnow</span>;</td></tr>
</table>
<!--TOC subsection id="sec6" Function mexTrainDL_Memory-->
<h3 class="subsection" id="sec6">3.2  Function mexTrainDL_Memory</h3><!--SEC END --><p>
Memory-consuming version of mexTrainDL. This function is well adapted to small/medium-size problems:
It requires storing all the coefficients α and is therefore impractical
for very large datasets. However, in many situations, one can afford this memory cost and it is better to use this method, which 
is faster than mexTrainDL.
Note that unlike mexTrainDL this function does not allow warm-restart.</p><table class="lstframe c011"><tr><td class="mouselstlisting"><span class="c001">% <br>
% Usage:   [D]=mexTrainDL(X,param);<br>
%<br>
% Name: mexTrainDL_Memory<br>
%<br>
% Description: mexTrainDL_Memory is an efficient but memory consuming <br>
%     variant of the dictionary learning technique presented in<br>
%<br>
%     "Online Learning for Matrix Factorization and Sparse Coding"<br>
%     by Julien Mairal, Francis Bach, Jean Ponce and Guillermo Sapiro<br>
%     arXiv:0908.0050<br>
%     <br>
%     "Online Dictionary Learning for Sparse Coding"      <br>
%     by Julien Mairal, Francis Bach, Jean Ponce and Guillermo Sapiro<br>
%     ICML 2009.<br>
%<br>
%     Contrary to the approaches above, the algorithm here <br>
%        does require to store all the coefficients from all the training<br>
%        signals. For this reason this variant can not be used with large<br>
%        training sets, but is more efficient than the regular online<br>
%        approach for training sets of reasonable size.<br>
%<br>
%     It addresses the dictionary learning problems<br>
%        1) if param.mode=1<br>
%     min_{D in C} (1/n) sum_{i=1}^n  ||alpha_i||_1  s.t.  ...<br>
%                                         ||x_i-Dalpha_i||_2^2 &lt;= lambda<br>
%        2) if param.mode=2<br>
%     min_{D in C} (1/n) sum_{i=1}^n (1/2)||x_i-Dalpha_i||_2^2 + ... <br>
%                                                      lambda||alpha_i||_1  <br>
%<br>
%     C is a convex set verifying<br>
%        1) if param.modeD=0<br>
%           C={  D in Real^{m x p}  s.t.  forall j,  ||d_j||_2^2 &lt;= 1 }<br>
%        1) if param.modeD=1<br>
%           C={  D in Real^{m x p}  s.t.  forall j,  ||d_j||_2^2 + ... <br>
%                                                  gamma1||d_j||_1 &lt;= 1 }<br>
%        1) if param.modeD=2<br>
%           C={  D in Real^{m x p}  s.t.  forall j,  ||d_j||_2^2 + ... <br>
%                                  gamma1||d_j||_1 + gamma2 FL(d_j) &lt;= 1 }<br>
%<br>
%     Potentially, n can be very large with this algorithm.<br>
%<br>
% Inputs: X:  double m x n matrix   (input signals)<br>
%               m is the signal size<br>
%               n is the number of signals to decompose<br>
%         param: struct<br>
%            param.D: (optional) double m x p matrix   (dictionary)<br>
%              p is the number of elements in the dictionary<br>
%              When D is not provided, the dictionary is initialized <br>
%              with random elements from the training set.<br>
%           param.K (size of the dictionary, optional is param.D is provided)<br>
%           param.lambda  (parameter)<br>
%           param.iter (number of iterations).  If a negative number is <br>
%              provided it will perform the computation during the<br>
%              corresponding number of seconds. For instance param.iter=-5<br>
%              learns the dictionary during 5 seconds.<br>
%            param.mode (optional, see above, by default 2) <br>
%            param.modeD (optional, see above, by default 0)<br>
%            param.posD (optional, adds positivity constraints on the <br>
%              dictionary, false by default, not compatible with <br>
%              param.modeD=2)<br>
%            param.gamma1 (optional parameter for param.modeD &gt;= 1)<br>
%            param.gamma2 (optional parameter for param.modeD = 2)<br>
%            param.batchsize (optional, size of the minibatch, by default <br>
%              512)<br>
%            param.iter_updateD (optional, number of BCD iterations for the dictionary <br>
%                update step, by default 1)<br>
%            param.modeParam (optimization mode).<br>
%              1) if param.modeParam=0, the optimization uses the <br>
%                 parameter free strategy of the ICML paper<br>
%              2) if param.modeParam=1, the optimization uses the <br>
%                 parameters rho as in arXiv:0908.0050<br>
%              3) if param.modeParam=2, the optimization uses exponential <br>
%                 decay weights with updates of the form <br>
%                 A_{t} &lt;- rho A_{t-1} + alpha_t alpha_t^T<br>
%            param.rho (optional) tuning parameter (see paper arXiv:0908.0050)<br>
%            param.t0 (optional) tuning parameter (see paper arXiv:0908.0050)<br>
%            param.clean (optional, true by default. prunes <br>
%              automatically the dictionary from unused elements).<br>
%            param.numThreads (optional, number of threads for exploiting<br>
%              multi-core / multi-cpus. By default, it takes the value -1,<br>
%              which automatically selects all the available CPUs/cores).<br>
%<br>
% Output: <br>
%         param.D: double m x p matrix   (dictionary)<br>
%<br>
% Note: this function admits a few experimental usages, which have not<br>
%     been extensively tested:<br>
%         - single precision setting (even though the output alpha is double <br>
%           precision)<br>
%<br>
% Author: Julien Mairal, 2009</span></td></tr>
</table><p>
The following piece of code illustrates how to use this function.
</p><table class="lstframe c011"><tr><td class="mouselstlisting"><span class="c002">clear</span> <span class="c002">all</span>;<br>
<br>
I=double(imread(<span class="c003">'data/lena.png'</span>))/255;<br>
<span class="c001">% extract 8 x 8 patches</span><br>
X=im2col(I,[8 8],<span class="c003">'sliding'</span>);<br>
X=X-repmat(<span class="c002">mean</span>(X),[<span class="c002">size</span>(X,1) 1]);<br>
X=X ./ repmat(<span class="c002">sqrt</span>(<span class="c002">sum</span>(X.^2)),[<span class="c002">size</span>(X,1) 1]);<br>
X=X(:,1:10:<span class="c002">end</span>);<br>
<br>
param.K=200;  <span class="c001">% learns a dictionary with 100 elements</span><br>
param.lambda=0.15;<br>
param.numThreads=4; <span class="c001">% number of threads</span><br>
<br>
param.iter=100;  <span class="c001">% let us see what happens after 100 iterations.<br>
<br>
%%%%%%%%%% FIRST EXPERIMENT %%%%%%%%%%%</span><br>
<span class="c002">tic</span><br>
D = mexTrainDL_Memory(X,param);<br>
t=<span class="c002">toc</span>;<br>
<span class="c002">fprintf</span>(<span class="c003">'time of computation for Dictionary Learning: %f\n'</span>,t);<br>
<br>
<span class="c002">fprintf</span>(<span class="c003">'Evaluating cost function...\n'</span>);<br>
alpha=mexLasso(X,D,param);<br>
R=<span class="c002">mean</span>(0.5*<span class="c002">sum</span>((X-D*alpha).^2)+param.lambda*<span class="c002">sum</span>(<span class="c002">abs</span>(alpha)));<br>
ImD=displayPatches(D);<br>
<span class="c002">subplot</span>(1,3,1);<br>
<span class="c002">imagesc</span>(ImD); <span class="c002">colormap</span>(<span class="c003">'gray'</span>);<br>
<span class="c002">fprintf</span>(<span class="c003">'objective function: %f\n'</span>,R);<br>
<br>
<span class="c001">%%%%%%%%%% SECOND EXPERIMENT %%%%%%%%%%%</span><br>
<span class="c002">tic</span><br>
D = mexTrainDL(X,param);<br>
t=<span class="c002">toc</span>;<br>
<span class="c002">fprintf</span>(<span class="c003">'time of computation for Dictionary Learning: %f\n'</span>,t);<br>
<br>
<span class="c002">fprintf</span>(<span class="c003">'Evaluating cost function...\n'</span>);<br>
alpha=mexLasso(X,D,param);<br>
R=<span class="c002">mean</span>(0.5*<span class="c002">sum</span>((X-D*alpha).^2)+param.lambda*<span class="c002">sum</span>(<span class="c002">abs</span>(alpha)));<br>
ImD=displayPatches(D);<br>
<span class="c002">subplot</span>(1,3,2);<br>
<span class="c002">imagesc</span>(ImD); <span class="c002">colormap</span>(<span class="c003">'gray'</span>);<br>
<span class="c002">fprintf</span>(<span class="c003">'objective function: %f\n'</span>,R);</td></tr>
</table>
<!--TOC subsection id="sec7" Function mexStructTrainDL-->
<h3 class="subsection" id="sec7">3.3  Function mexStructTrainDL</h3><!--SEC END --><p>
This function allows to use mexTrainDL with structured regularization functions
for the coefficients α. It internally uses the FISTA algorithm.
</p><table class="lstframe c011"><tr><td class="mouselstlisting"><span class="c001">% <br>
% Usage:   [D [model]]=mexStructTrainDL(X,param[,model]);<br>
%          model is optional<br>
%<br>
% Name: mexStructTrainDL<br>
%<br>
% Description: mexStructTrainDL is an efficient implementation of the<br>
%     dictionary learning technique presented in<br>
%<br>
%     "Online Learning for Matrix Factorization and Sparse Coding"<br>
%     by Julien Mairal, Francis Bach, Jean Ponce and Guillermo Sapiro<br>
%     arXiv:0908.0050<br>
%     <br>
%     "Online Dictionary Learning for Sparse Coding"      <br>
%     by Julien Mairal, Francis Bach, Jean Ponce and Guillermo Sapiro<br>
%     ICML 2009.<br>
%<br>
%<br>
%     It addresses the dictionary learning problems<br>
%        min_{D in C} (1/n) sum_{i=1}^n 0.5||x_i-Dalpha_i||_2^2 + lambda psi(alpha)<br>
%        where the regularization function psi depends on param.regul<br>
%        (see mexProximalFlat for the description of psi,<br>
%         and param.regul below for allowed values of regul)<br>
%<br>
%%     C is a convex set verifying<br>
%        1) if param.modeD=0<br>
%           C={  D in Real^{m x p}  s.t.  forall j,  ||d_j||_2^2 &lt;= 1 }<br>
%        2) if param.modeD=1<br>
%           C={  D in Real^{m x p}  s.t.  forall j,  ||d_j||_2^2 + ... <br>
%                                                  gamma1||d_j||_1 &lt;= 1 }<br>
%        3) if param.modeD=2<br>
%           C={  D in Real^{m x p}  s.t.  forall j,  ||d_j||_2^2 + ... <br>
%                                  gamma1||d_j||_1 + gamma2 FL(d_j) &lt;= 1 }<br>
%        4) if param.modeD=3<br>
%           C={  D in Real^{m x p}  s.t.  forall j,  (1-gamma1)||d_j||_2^2 + ... <br>
%                                  gamma1||d_j||_1 &lt;= 1 }<br>
<br>
%<br>
%     Potentially, n can be very large with this algorithm.<br>
%<br>
% Inputs: X:  double m x n matrix   (input signals)<br>
%               m is the signal size<br>
%               n is the number of signals to decompose<br>
%         param: struct<br>
%            param.D: (optional) double m x p matrix   (dictionary)<br>
%              p is the number of elements in the dictionary<br>
%              When D is not provided, the dictionary is initialized <br>
%              with random elements from the training set.<br>
%           param.K (size of the dictionary, optional is param.D is provided)<br>
%           param.lambda  (parameter)<br>
%           param.lambda2  (optional, by default 0)<br>
%           param.lambda3 (optional, regularization parameter, 0 by default)<br>
%           param.iter (number of iterations).  If a negative number is <br>
%              provided it will perform the computation during the<br>
%              corresponding number of seconds. For instance param.iter=-5<br>
%              learns the dictionary during 5 seconds.<br>
%           param.regul choice of regularization : one of<br>
%               'l0' 'l1' 'l2' 'linf' 'none' 'elastic-net' 'fused-lasso'<br>
%               'graph' 'graph-ridge' 'graph-l2' 'tree-l0' 'tree-l2' 'tree-linf' <br>
%           param.tree struct (see documentation of mexProximalTree);<br>
%               needed for param.regul of graph kind.<br>
%           param.graph struct (see documentation of mexProximalGraph);<br>
%               needed for param.regul of tree kind.<br>
%           param.posAlpha (optional, adds positivity constraints on the<br>
%               coefficients, false by default.<br>
%           param.modeD (optional, see above, by default 0)<br>
%           param.posD (optional, adds positivity constraints on the <br>
%             dictionary, false by default, not compatible with <br>
%             param.modeD=2)<br>
%           param.gamma1 (optional parameter for param.modeD &gt;= 1)<br>
%           param.gamma2 (optional parameter for param.modeD = 2)<br>
%           param.batchsize (optional, size of the minibatch, by default <br>
%              512)<br>
%           param.iter_updateD (optional, number of BCD iterations for the dictionary<br>
%              update step, by default 1)<br>
%           param.modeParam (optimization mode).<br>
%              1) if param.modeParam=0, the optimization uses the <br>
%                 parameter free strategy of the ICML paper<br>
%              2) if param.modeParam=1, the optimization uses the <br>
%                 parameters rho as in arXiv:0908.0050<br>
%              3) if param.modeParam=2, the optimization uses exponential <br>
%                 decay weights with updates of the form <br>
%                 A_{t} &lt;- rho A_{t-1} + alpha_t alpha_t^T<br>
%            param.ista (optional, use ista instead of fista, false by default).<br>
%            param.tol (optional, tolerance for stopping criteration, which is a relative duality gap<br>
%            param.fixed_step (deactive the line search for L in fista and use param.K instead)<br>
%           param.rho (optional) tuning parameter (see paper arXiv:0908.0050)<br>
%           param.t0 (optional) tuning parameter (see paper arXiv:0908.0050)<br>
%           param.clean (optional, true by default. prunes <br>
%              automatically the dictionary from unused elements).<br>
%           param.verbose (optional, true by default, increase verbosity)<br>
%           param.numThreads (optional, number of threads for exploiting<br>
%              multi-core / multi-cpus. By default, it takes the value -1,<br>
%              which automatically selects all the available CPUs/cores).<br>
%<br>
% Output: <br>
%         param.D: double m x p matrix   (dictionary)<br>
%<br>
% Note: this function admits a few experimental usages, which have not<br>
%     been extensively tested:<br>
%         - single precision setting <br>
%<br>
% Author: Julien Mairal, 2009</span></td></tr>
</table><p>
The following piece of code illustrates how to use this function.
</p><table class="lstframe c011"><tr><td class="mouselstlisting"><span class="c002">clear</span> <span class="c002">all</span>;<br>
<br>
I=double(imread(<span class="c003">'data/lena.png'</span>))/255;<br>
<span class="c001">% extract 8 x 8 patches</span><br>
X=im2col(I,[8 8],<span class="c003">'sliding'</span>);<br>
X=X-repmat(<span class="c002">mean</span>(X),[<span class="c002">size</span>(X,1) 1]);<br>
X=X ./ repmat(<span class="c002">sqrt</span>(<span class="c002">sum</span>(X.^2)),[<span class="c002">size</span>(X,1) 1]);<br>
<br>
param.K=64;  <span class="c001">% learns a dictionary with 64 elements</span><br>
param.lambda=0.05;<br>
param.numThreads=4; <span class="c001">% number of threads</span><br>
param.batchsize=400;<br>
param.tol = 1e-3<br>
<br>
param.iter=200;  <span class="c001">%</span><br>
<br>
<span class="c002">if</span> false<br>
param.regul = <span class="c003">'l1'</span>;<br>
<span class="c002">fprintf</span>(<span class="c003">'with Fista Regression %s\n'</span>,param.regul);<br>
<span class="c002">tic</span><br>
D = mexStructTrainDL(X,param);<br>
t=<span class="c002">toc</span>;<br>
<span class="c002">fprintf</span>(<span class="c003">'time of computation for Dictionary Learning: %f\n'</span>,t);<br>
<span class="c001">%</span><br>
param.regul = <span class="c003">'l2'</span>;<br>
<span class="c002">fprintf</span>(<span class="c003">'with Fista Regression %s\n'</span>,param.regul);<br>
<span class="c002">tic</span><br>
D = mexStructTrainDL(X,param);<br>
t=<span class="c002">toc</span>;<br>
<span class="c002">fprintf</span>(<span class="c003">'time of computation for Dictionary Learning: %f\n'</span>,t);<br>
<span class="c001">%</span><br>
param.regul = <span class="c003">'elastic-net'</span>;<br>
<span class="c002">fprintf</span>(<span class="c003">'with Fista %s\n'</span>,param.regul);<br>
param.lambda2=0.1;<br>
<span class="c002">tic</span><br>
D = mexStructTrainDL(X,param);<br>
t=<span class="c002">toc</span>;<br>
<span class="c002">fprintf</span>(<span class="c003">'time of computation for Dictionary Learning: %f\n'</span>,t);<br>
<br>
<span class="c001">%%% GRAPH</span><br>
param.lambda=0.1; <span class="c001">% regularization parameter</span><br>
param.tol=1e-5;<br>
param.K = 10<br>
graph.eta_g=[1 1 1 1 1];<br>
graph.groups=<span class="c002">sparse</span>([0 0 0 1 0;<br>
                     0 0 0 0 0;<br>
                     0 0 0 0 0;<br>
                     0 0 0 0 0;<br>
                     0 0 1 0 0]);   <span class="c001">% g5 is included in g3, and g2 is included in g4</span><br>
graph.groups_var=<span class="c002">sparse</span>([1 0 0 0 0;<br>
                         1 0 0 0 0;<br>
                         1 0 0 0 0 ;<br>
                         1 1 0 0 0;<br>
                         0 1 0 1 0;<br>
                         0 1 0 1 0;<br>
                         0 1 0 0 1;<br>
                         0 0 0 0 1;<br>
                         0 0 0 0 1;<br>
                         0 0 1 0 0]); <span class="c001">% represents direct inclusion relations </span><br>
<br>
param.graph = graph<br>
<br>
param.regul = <span class="c003">'graph'</span>;<br>
<span class="c002">fprintf</span>(<span class="c003">'with Fista %s\n'</span>,param.regul);<br>
<span class="c002">tic</span><br>
D = mexStructTrainDL(X,param);<br>
t=<span class="c002">toc</span>;<br>
<span class="c002">fprintf</span>(<span class="c003">'time of computation for Dictionary Learning: %f\n'</span>,t);<br>
<br>
<span class="c001">%%<br>
%%% TREE<br>
%?pause;</span><br>
param = rmfield(param,<span class="c003">'graph'</span>);<br>
<br>
<span class="c002">end</span><br>
<br>
<br>
param.lambda=0.1; <span class="c001">% regularization parameter</span><br>
param.tol=1e-5;<br>
param.K = 10<br>
<br>
tree.own_variables=  int32([0 0 3 5 6 6 8 9]);   <span class="c001">% pointer to the first variable of each group</span><br>
tree.N_own_variables=int32([0 3 2 1 0 2 1 1]); <span class="c001">% number of "root" variables in each group</span><br>
tree.eta_g=[1 1 1 2 2 2 2.5 2.5];<br>
tree.groups=<span class="c002">sparse</span>([0 0 0 0 0 0 0 0; ...<br>
                    1 0 0 0 0 0 0 0; ...<br>
                    0 1 0 0 0 0 0 0; ...<br>
                    0 1 0 0 0 0 0 0; ...<br>
                    1 0 0 0 0 0 0 0; ...<br>
                    0 0 0 0 1 0 0 0; ...<br>
                    0 0 0 0 1 0 0 0; ...<br>
                    0 0 0 0 0 0 1 0]);  <span class="c001">% first group should always be the root of the tree</span><br>
<br>
param.tree = tree;<br>
<br>
param.regul = <span class="c003">'tree-l0'</span>;<br>
<span class="c002">fprintf</span>(<span class="c003">'with Fista %s\n'</span>,param.regul);<br>
<br>
<span class="c002">tic</span><br>
D = mexStructTrainDL(X,param);<br>
t=<span class="c002">toc</span>;<br>
<span class="c002">fprintf</span>(<span class="c003">'time of computation for Dictionary Learning: %f\n'</span>,t);<br>
<br>
<span class="c001">%</span><br>
param.regul = <span class="c003">'tree-l2'</span>;<br>
<span class="c002">fprintf</span>(<span class="c003">'with Fista %s\n'</span>,param.regul);<br>
<span class="c002">tic</span><br>
D = mexStructTrainDL(X,param);<br>
t=<span class="c002">toc</span>;<br>
<span class="c002">fprintf</span>(<span class="c003">'time of computation for Dictionary Learning: %f\n'</span>,t);<br>
<br>
<span class="c001">%</span><br>
param.regul = <span class="c003">'tree-linf'</span>;<br>
<span class="c002">fprintf</span>(<span class="c003">'with Fista %s\n'</span>,param.regul);<br>
<br>
<span class="c002">tic</span><br>
D = mexStructTrainDL(X,param);<br>
t=<span class="c002">toc</span>;<br>
<span class="c002">fprintf</span>(<span class="c003">'time of computation for Dictionary Learning: %f\n'</span>,t);</td></tr>
</table>
<!--TOC subsection id="sec8" Function nmf-->
<h3 class="subsection" id="sec8">3.4  Function nmf</h3><!--SEC END --><p>
This function is an example on how to use the function mexTrainDL for the
problem of non-negative matrix factorization formulated in [<a href="#lee2">17</a>]. Note
that mexTrainDL can be replaced by mexTrainDL_Memory in this function for
small or medium datasets.</p><table class="lstframe c011"><tr><td class="mouselstlisting"><span class="c001">% <br>
% Usage:   [U [,V]]=nmf(X,param);<br>
%<br>
% Name: nmf<br>
%<br>
% Description: mexTrainDL is an efficient implementation of the<br>
%     non-negative matrix factorization technique presented in <br>
%<br>
%     "Online Learning for Matrix Factorization and Sparse Coding"<br>
%     by Julien Mairal, Francis Bach, Jean Ponce and Guillermo Sapiro<br>
%     arXiv:0908.0050<br>
%     <br>
%     "Online Dictionary Learning for Sparse Coding"      <br>
%     by Julien Mairal, Francis Bach, Jean Ponce and Guillermo Sapiro<br>
%     ICML 2009.<br>
%<br>
%     Potentially, n can be very large with this algorithm.<br>
%<br>
% Inputs: X:  double m x n matrix   (input signals)<br>
%               m is the signal size<br>
%               n is the number of signals to decompose<br>
%         param: struct<br>
%            param.K (number of required factors)<br>
%            param.iter (number of iterations).  If a negative number <br>
%              is provided it will perform the computation during the<br>
%              corresponding number of seconds. For instance param.iter=-5<br>
%              learns the dictionary during 5 seconds.<br>
%            param.batchsize (optional, size of the minibatch, by default <br>
%               512)<br>
%            param.modeParam (optimization mode).<br>
%               1) if param.modeParam=0, the optimization uses the <br>
%                  parameter free strategy of the ICML paper<br>
%               2) if param.modeParam=1, the optimization uses the <br>
%                  parameters rho as in arXiv:0908.0050<br>
%               3) if param.modeParam=2, the optimization uses exponential <br>
%                  decay weights with updates of the form  <br>
%                  A_{t} &lt;- rho A_{t-1} + alpha_t alpha_t^T<br>
%            param.rho (optional) tuning parameter (see paper <br>
%              arXiv:0908.0050)<br>
%            param.t0 (optional) tuning parameter (see paper <br>
%              arXiv:0908.0050)<br>
%            param.clean (optional, true by default. prunes automatically <br>
%              the dictionary from unused elements).<br>
%            param.batch (optional, false by default, use batch learning <br>
%              instead of online learning)<br>
%            param.numThreads (optional, number of threads for exploiting<br>
%                 multi-core / multi-cpus. By default, it takes the value -1,<br>
%                 which automatically selects all the available CPUs/cores).<br>
%         model: struct (optional) learned model for "retraining" the data.<br>
%<br>
% Output:<br>
%         U: double m x p matrix   <br>
%         V: double p x n matrix   (optional)<br>
%         model: struct (optional) learned model to be used for <br>
%           "retraining" the data.<br>
%<br>
% Author: Julien Mairal, 2009</span><br>
<span class="c002">function</span> [U V] = nmf(X,param)<br>
<br>
param.lambda=0;<br>
param.mode=2;<br>
param.posAlpha=1;<br>
param.posD=1;<br>
param.whiten=0;<br>
U=mexTrainDL(X,param);<br>
param.pos=1;<br>
<span class="c002">if</span> <span class="c002">nargout</span> == 2<br>
   <span class="c002">if</span> <span class="c002">issparse</span>(X) <span class="c001">% todo allow sparse matrices X for mexLasso</span><br>
      maxbatch=<span class="c002">ceil</span>(10000000/<span class="c002">size</span>(X,1));<br>
      <span class="c002">for</span> jj = 1:maxbatch:<span class="c002">size</span>(X,2)<br>
         indbatch=jj:<span class="c002">min</span>((jj+maxbatch-1),<span class="c002">size</span>(X,2));<br>
         Xb=<span class="c002">full</span>(X(:,indbatch));<br>
         V(:,indbatch)=mexLasso(Xb,U,param);<br>
      <span class="c002">end</span><br>
   <span class="c002">else</span><br>
      V=mexLasso(X,U,param);<br>
   <span class="c002">end<br>
end</span></td></tr>
</table><p>
The following piece of code illustrates how to use this function.
</p><table class="lstframe c011"><tr><td class="mouselstlisting"><span class="c002">clear</span> <span class="c002">all</span>;<br>
<br>
I=double(imread(<span class="c003">'data/lena.png'</span>))/255;<br>
<span class="c001">% extract 8 x 8 patches</span><br>
X=im2col(I,[16 16],<span class="c003">'sliding'</span>);<br>
X=X(:,1:10:<span class="c002">end</span>);<br>
X=X ./ repmat(<span class="c002">sqrt</span>(<span class="c002">sum</span>(X.^2)),[<span class="c002">size</span>(X,1) 1]);<br>
<br>
param.K=49;  <span class="c001">% learns a dictionary with 100 elements</span><br>
param.numThreads=4; <span class="c001">% number of threads</span><br>
<br>
param.iter=-5;  <span class="c001">% let us see what happens after 100 iterations.<br>
<br>
%%%%%%%%%% FIRST EXPERIMENT %%%%%%%%%%%</span><br>
<span class="c002">tic</span><br>
[U V] = nmf(X,param);<br>
t=<span class="c002">toc</span>;<br>
<span class="c002">fprintf</span>(<span class="c003">'time of computation for Dictionary Learning: %f\n'</span>,t);<br>
<br>
<span class="c002">fprintf</span>(<span class="c003">'Evaluating cost function...\n'</span>);<br>
R=<span class="c002">mean</span>(0.5*<span class="c002">sum</span>((X-U*V).^2));<br>
ImD=displayPatches(U);<br>
<span class="c002">imagesc</span>(ImD); <span class="c002">colormap</span>(<span class="c003">'gray'</span>);<br>
<span class="c002">fprintf</span>(<span class="c003">'objective function: %f\n'</span>,R);</td></tr>
</table>
<!--TOC subsection id="sec9" Function nnsc-->
<h3 class="subsection" id="sec9">3.5  Function nnsc</h3><!--SEC END --><p>
This function is an example on how to use the function mexTrainDL for the
problem of non-negative sparse coding as defined in [<a href="#hoyer">14</a>]. Note that
mexTrainDL can be replaced by mexTrainDL_Memory in this function for small or
medium datasets.</p><table class="lstframe c011"><tr><td class="mouselstlisting"><span class="c001">% <br>
% Usage:   [U [,V]]=nnsc(X,param);<br>
%<br>
% Name: nmf<br>
%<br>
% Description: mexTrainDL is an efficient implementation of the<br>
%     non-negative sparse coding technique presented in <br>
%<br>
%     "Online Learning for Matrix Factorization and Sparse Coding"<br>
%     by Julien Mairal, Francis Bach, Jean Ponce and Guillermo Sapiro<br>
%     arXiv:0908.0050<br>
%     <br>
%     "Online Dictionary Learning for Sparse Coding"      <br>
%     by Julien Mairal, Francis Bach, Jean Ponce and Guillermo Sapiro<br>
%     ICML 2009.<br>
%<br>
%     Potentially, n can be very large with this algorithm.<br>
%<br>
% Inputs: X:  double m x n matrix   (input signals)<br>
%               m is the signal size<br>
%               n is the number of signals to decompose<br>
%         param: struct<br>
%           param.K (number of required factors)<br>
%           param.lambda (parameter)<br>
%           param.iter (number of iterations).  If a negative number <br>
%              is provided it will perform the computation during the<br>
%              corresponding number of seconds. For instance param.iter=-5<br>
%              learns the dictionary during 5 seconds.<br>
%           param.batchsize (optional, size of the minibatch, by default <br>
%              512)<br>
%           param.modeParam (optimization mode).<br>
%              1) if param.modeParam=0, the optimization uses the <br>
%                 parameter free strategy of the ICML paper<br>
%              2) if param.modeParam=1, the optimization uses the <br>
%                 parameters rho as in arXiv:0908.0050<br>
%              3) if param.modeParam=2, the optimization uses exponential <br>
%                 decay weights with updates of the form <br>
%                 A_{t} &lt;- rho A_{t-1} + alpha_t alpha_t^T<br>
%           param.rho (optional) tuning parameter (see paper<br>
%              arXiv:0908.0050)<br>
%           param.t0 (optional) tuning parameter (see paper <br>
%              arXiv:0908.0050)<br>
%           param.clean (optional, true by default. prunes automatically <br>
%              the dictionary from unused elements).<br>
%           param.batch (optional, false by default, use batch learning <br>
%              instead of online learning)<br>
%           param.numThreads (optional, number of threads for exploiting<br>
%              multi-core / multi-cpus. By default, it takes the value -1,<br>
%              which automatically selects all the available CPUs/cores).<br>
%         model: struct (optional) learned model for "retraining" the data.<br>
%<br>
% Output:<br>
%         U: double m x p matrix   <br>
%         V: double p x n matrix   (optional)<br>
%         model: struct (optional) learned model to be used for <br>
%            "retraining" the data.<br>
%<br>
% Author: Julien Mairal, 2009</span><br>
<span class="c002">function</span> [U V] = nnsc(X,param)<br>
<br>
param.mode=2;<br>
param.posAlpha=1;<br>
param.posD=1;<br>
param.whiten=0;<br>
U=mexTrainDL(X,param);<br>
param.pos=1;<br>
<span class="c002">if</span> <span class="c002">nargout</span> == 2<br>
   V=mexLasso(X,U,param);<br>
<span class="c002">end</span></td></tr>
</table>
<!--TOC subsection id="sec10" Function mexArchetypalAnalysis-->
<h3 class="subsection" id="sec10">3.6  Function mexArchetypalAnalysis</h3><!--SEC END --><p>
The function optimizes the archetypal analysis formulation of [<a href="#Cut94">7</a>]. It follows the
methodology presented in the following paper [<a href="#ChenCVPR">37</a>].</p><table class="lstframe c011"><tr><td class="mouselstlisting"><span class="c001">% <br>
% Usage:  [Z [A,B]]=mexArchetypalAnalysis(X,param);<br>
% Usage:  [Z [A,B]]=mexArchetypalAnalysis(X,Z,param);<br>
%<br>
% Name: mexArchetypalAnalysis<br>
%<br>
% documentation to appear soon<br>
%<br>
% Inputs: X:  double m x n matrix   (input signals)<br>
%               m is the signal size<br>
%               n is the number of signals to decompose<br>
% Output: Z: double %<br>
%<br>
% Author: Yuansi Chen and Julien Mairal, 2014</span></td></tr>
</table><p>
The following piece of code illustrates how to use this function.
</p><table class="lstframe c011"><tr><td class="mouselstlisting"><span class="c002">clear</span> <span class="c002">all</span>;<br>
<span class="c002">rand</span>(<span class="c003">'seed'</span>,0);<br>
I=double(imread(<span class="c003">'data/lena.png'</span>))/255;<br>
<span class="c001">% extract 8 x 8 patches</span><br>
X=im2col(I,[8 8],<span class="c003">'sliding'</span>);<br>
per=<span class="c002">randperm</span>(<span class="c002">size</span>(X,2));<br>
X=X(:,per(1:10000));<br>
X=X-repmat(<span class="c002">mean</span>(X),[<span class="c002">size</span>(X,1) 1]);<br>
nrm=<span class="c002">sqrt</span>(<span class="c002">sum</span>(X.^2));<br>
me=<span class="c002">median</span>(nrm);<br>
X=X ./ repmat(<span class="c002">max</span>(nrm,me),[<span class="c002">size</span>(X,1) 1]);<br>
<br>
param.p=64;  <span class="c001">% learns a dictionary with 64 elements</span><br>
param.robust=false;<br>
param.epsilon=1e-3;  <span class="c001">% width for Huber loss</span><br>
param.computeXtX=true;<br>
param.stepsFISTA=0;<br>
param.stepsAS=10;<br>
param.numThreads=-1;<br>
<br>
<span class="c001">%%%%%%%%%% FIRST EXPERIMENT %%%%%%%%%%%</span><br>
<span class="c002">tic</span><br>
[Z A B] = mexArchetypalAnalysis(X,param);<br>
t=<span class="c002">toc</span>;<br>
<span class="c002">fprintf</span>(<span class="c003">'time of computation for Archetypal Analysis: %f\n'</span>,t);<br>
R=<span class="c002">norm</span>(X-Z*A,<span class="c003">'fro'</span>)^2;<br>
<span class="c002">fprintf</span>(<span class="c003">'objective function: %f\n'</span>,R);<br>
<br>
<span class="c002">fprintf</span>(<span class="c003">'Re-Evaluating cost function after re-updating A...\n'</span>);<br>
paramdecomp.computeXtX=true;<br>
paramdecomp.numThreads=-1;<br>
A2=mexDecompSimplex(X,Z,param);<br>
R=<span class="c002">norm</span>(X-Z*A2,<span class="c003">'fro'</span>)^2;<br>
<span class="c002">fprintf</span>(<span class="c003">'objective function: %f\n'</span>,R);<br>
<br>
<span class="c001">%%%%%%%%%% Second EXPERIMENT %%%%%%%%%%%</span><br>
<span class="c002">tic</span><br>
[Z2 A2] = mexArchetypalAnalysis(X,param,Z);<br>
t=<span class="c002">toc</span>;<br>
<span class="c002">fprintf</span>(<span class="c003">'time of computation for Archetypal Analysis: %f\n'</span>,t);<br>
<span class="c002">fprintf</span>(<span class="c003">'Evaluating cost function...\n'</span>);<br>
R=<span class="c002">norm</span>(X-Z2*A2,<span class="c003">'fro'</span>)^2;<br>
<span class="c002">fprintf</span>(<span class="c003">'objective function: %f\n'</span>,R);<br>
<br>
<span class="c001">%%%%%%%%%% THIRD EXPERIMENT %%%%%%%%%%%</span><br>
<span class="c002">tic</span><br>
param.robust=true;<br>
[Z3] = mexArchetypalAnalysis(X,param);<br>
t=<span class="c002">toc</span>;<br>
<span class="c002">fprintf</span>(<span class="c003">'time of computation for Robust Archetypal Analysis: %f\n'</span>,t);</td></tr>
</table>
<!--TOC section id="sec11" Sparse Decomposition Toolbox-->
<h2 class="section" id="sec11">4  Sparse Decomposition Toolbox</h2><!--SEC END --><p>
This toolbox implements several algorithms for solving signal reconstruction problems. It is mostly adapted for solving a large number of small/medium scale problems, but can be also efficient sometimes with large scale ones.
</p>
<!--TOC subsection id="sec12" Function mexOMP-->
<h3 class="subsection" id="sec12">4.1  Function mexOMP</h3><!--SEC END --><p>
This is a fast implementation of the Orthogonal Matching Pursuit algorithm (or forward selection) [<a href="#mallat4">27</a>, <a href="#weisberg">35</a>]. Given a matrix of signals <span class="c009">X</span>=[<span class="c009">x</span><sup>1</sup>,…,<span class="c009">x</span><sup><span class="c007">n</span></sup>] in ℝ<sup><span class="c007">m</span> × <span class="c007">n</span></sup> and a dictionary <span class="c009">D</span>=[<span class="c009">d</span><sup>1</sup>,…,<span class="c009">d</span><sup><span class="c007">p</span></sup>] in ℝ<sup><span class="c007">m</span> × <span class="c007">p</span></sup>, the algorithm computes a matrix <span class="c009">A</span>=[α<sup>1</sup>,…,α<sup><span class="c007">n</span></sup>] in ℝ<sup><span class="c007">p</span> × <span class="c007">n</span></sup>,
where for each column <span class="c009">x</span> of <span class="c009">X</span>, it returns a coefficient vector α which is an approximate solution of the following NP-hard problem
</p><table class="display dcenter"><tr class="c016"><td class="dcell">
</td><td class="dcell"><table class="display"><tr><td class="dcell c012">&nbsp;</td></tr>
<tr><td class="dcell c012">min</td></tr>
<tr><td class="dcell c012">α ∈ ℝ<sup><span class="c007">p</span></sup></td></tr>
</table></td><td class="dcell"> ||<span class="c009">x</span>−<span class="c009">D</span>α||<sub>2</sub><sup>2</sup>   s.t.   ||α||<sub>0</sub> ≤ <span class="c007">L</span>,
    (2)</td></tr>
</table><p>
or 
</p><table class="display dcenter"><tr class="c016"><td class="dcell">
</td><td class="dcell"><table class="display"><tr><td class="dcell c012">&nbsp;</td></tr>
<tr><td class="dcell c012">min</td></tr>
<tr><td class="dcell c012">α ∈ ℝ<sup><span class="c007">p</span></sup></td></tr>
</table></td><td class="dcell">  ||α||<sub>0</sub>   s.t.   ||<span class="c009">x</span>−<span class="c009">D</span>α||<sub>2</sub><sup>2</sup> ≤ ε,
    (3)</td></tr>
</table><p>
or
</p><table class="display dcenter"><tr class="c016"><td class="dcell">
</td><td class="dcell"><table class="display"><tr><td class="dcell c012">&nbsp;</td></tr>
<tr><td class="dcell c012">min</td></tr>
<tr><td class="dcell c012">α ∈ ℝ<sup><span class="c007">p</span></sup></td></tr>
</table></td><td class="dcell"> </td><td class="dcell"><table class="display"><tr><td class="dcell c012">1</td></tr>
<tr><td class="hbar"></td></tr>
<tr><td class="dcell c012">2</td></tr>
</table></td><td class="dcell">||<span class="c009">x</span>−<span class="c009">D</span>α||<sub>2</sub><sup>2</sup> + λ ||α||<sub>0</sub>.
    (4)</td></tr>
</table><p>
For efficienty reasons, the method first computes the covariance matrix
<span class="c009">D</span><sup><span class="c007">T</span></sup><span class="c009">D</span>, then for each signal, it computes <span class="c009">D</span><sup><span class="c007">T</span></sup><span class="c009">x</span> and performs the
decomposition with a Cholesky-based algorithm (see [<a href="#cotter">6</a>] for instance).</p><p>Note that mexOMP can return the “greedy” regularization path if needed (see below):
</p><table class="lstframe c011"><tr><td class="mouselstlisting"><span class="c001">% <br>
% Usage:   A=mexOMP(X,D,param);<br>
% or       [A path]=mexOMP(X,D,param);<br>
%<br>
% Name: mexOMP<br>
%<br>
% Description: mexOMP is an efficient implementation of the<br>
%     Orthogonal Matching Pursuit algorithm. It is optimized<br>
%     for solving a large number of small or medium-sized <br>
%     decomposition problem (and not for a single large one).<br>
%     It first computes the Gram matrix D'D and then perform<br>
%     a Cholesky-based OMP of the input signals in parallel.<br>
%     X=[x^1,...,x^n] is a matrix of signals, and it returns<br>
%     a matrix A=[alpha^1,...,alpha^n] of coefficients.<br>
%     <br>
%     it addresses for all columns x of X, <br>
%         min_{alpha} ||alpha||_0  s.t  ||x-Dalpha||_2^2 &lt;= eps<br>
%         or<br>
%         min_{alpha} ||x-Dalpha||_2^2  s.t. ||alpha||_0 &lt;= L<br>
%         or<br>
%         min_{alpha} 0.5||x-Dalpha||_2^2 + lambda||alpha||_0 <br>
%         <br>
%<br>
% Inputs: X:  double m x n matrix   (input signals)<br>
%            m is the signal size<br>
%            n is the number of signals to decompose<br>
%         D:  double m x p matrix   (dictionary)<br>
%            p is the number of elements in the dictionary<br>
%            All the columns of D should have unit-norm !<br>
%         param: struct<br>
%            param.L (optional, maximum number of elements in each decomposition, <br>
%               min(m,p) by default)<br>
%            param.eps (optional, threshold on the squared l2-norm of the residual,<br>
%               0 by default<br>
%            param.lambda (optional, penalty parameter, 0 by default<br>
%            param.numThreads (optional, number of threads for exploiting<br>
%            multi-core / multi-cpus. By default, it takes the value -1,<br>
%            which automatically selects all the available CPUs/cores).<br>
%<br>
% Output: A: double sparse p x n matrix (output coefficients)<br>
%         path (optional): double dense p x L matrix (regularization path of the first signal)<br>
%<br>
% Note: this function admits a few experimental usages, which have not<br>
%     been extensively tested:<br>
%      - single precision setting (even though the output alpha is double <br>
%        precision)<br>
%      - Passing an int32 vector of length n to param.L provides<br>
%        a different parameter L for each input signal x_i<br>
%      - Passing a double vector of length n to param.eps and or param.lambda <br>
%        provides a different parameter eps (or lambda) for each input signal x_i<br>
%<br>
% Author: Julien Mairal, 2009</span></td></tr>
</table><p>
The following piece of code illustrates how to use this function.
</p><table class="lstframe c011"><tr><td class="mouselstlisting"><span class="c002">clear</span> <span class="c002">all</span>;<br>
<span class="c002">randn</span>(<span class="c003">'seed'</span>,0);<br>
<br>
<span class="c002">fprintf</span>(<span class="c003">'test mexOMP'</span>);<br>
<br>
<span class="c001">%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%<br>
% Decomposition of a large number of signals<br>
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%</span><br>
X=<span class="c002">randn</span>(64,100000);<br>
D=<span class="c002">randn</span>(64,200);<br>
D=D./repmat(<span class="c002">sqrt</span>(<span class="c002">sum</span>(D.^2)),[<span class="c002">size</span>(D,1) 1]);<br>
<span class="c001">% parameter of the optimization procedure are chosen</span><br>
param.L=10; <span class="c001">% not more than 10 non-zeros coefficients</span><br>
param.<span class="c002">eps</span>=0.1; <span class="c001">% squared norm of the residual should be less than 0.1</span><br>
param.numThreads=-1; <span class="c001">% number of processors/cores to use; the default choice is -1</span><br>
                    <span class="c001">% and uses all the cores of the machine</span><br>
<span class="c002">tic</span><br>
alpha=mexOMP(X,D,param);<br>
t=<span class="c002">toc<br>
fprintf</span>(<span class="c003">'%f signals processed per second\n'</span>,<span class="c002">size</span>(X,2)/t);<br>
<br>
<span class="c001">%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%<br>
% Regularization path of a single signal <br>
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%</span><br>
X=<span class="c002">randn</span>(64,1);<br>
D=<span class="c002">randn</span>(64,10);<br>
param.L=5;<br>
D=D./repmat(<span class="c002">sqrt</span>(<span class="c002">sum</span>(D.^2)),[<span class="c002">size</span>(D,1) 1]);<br>
[alpha <span class="c002">path</span>]=mexOMP(X,D,param);</td></tr>
</table>
<!--TOC subsection id="sec13" Function mexOMPMask-->
<h3 class="subsection" id="sec13">4.2  Function mexOMPMask</h3><!--SEC END --><p>
This is a variant of mexOMP with the possibility of handling a binary mask. 
Given a binary mask <span class="c009">B</span>=[β<sup>1</sup>,…,β<sup><span class="c007">n</span></sup>] in {0,1}<sup><span class="c007">m</span> × <span class="c007">n</span></sup>, it returns a matrix <span class="c009">A</span>=[α<sup>1</sup>,…,α<sup><span class="c007">n</span></sup>] such that for every column <span class="c009">x</span> of <span class="c009">X</span>, β of <span class="c009">B</span>, it computes a column α of <span class="c009">A</span> by addressing
</p><table class="display dcenter"><tr class="c016"><td class="dcell">
</td><td class="dcell"><table class="display"><tr><td class="dcell c012">&nbsp;</td></tr>
<tr><td class="dcell c012">min</td></tr>
<tr><td class="dcell c012">α ∈ ℝ<sup><span class="c007">p</span></sup></td></tr>
</table></td><td class="dcell"> ||diag(β)(<span class="c009">x</span>−<span class="c009">D</span>α)||<sub>2</sub><sup>2</sup>   s.t.   ||α||<sub>0</sub> ≤ <span class="c007">L</span>,
    (5)</td></tr>
</table><p>
or 
</p><table class="display dcenter"><tr class="c016"><td class="dcell">
</td><td class="dcell"><table class="display"><tr><td class="dcell c012">&nbsp;</td></tr>
<tr><td class="dcell c012">min</td></tr>
<tr><td class="dcell c012">α ∈ ℝ<sup><span class="c007">p</span></sup></td></tr>
</table></td><td class="dcell">  ||α||<sub>0</sub>   s.t.   ||diag(β)(<span class="c009">x</span>−<span class="c009">D</span>α)||<sub>2</sub><sup>2</sup> ≤ ε</td><td class="dcell"><table class="display"><tr><td class="dcell c012">||β||<sub>0</sub></td></tr>
<tr><td class="hbar"></td></tr>
<tr><td class="dcell c012"><span class="c007">m</span></td></tr>
</table></td><td class="dcell">,
    (6)</td></tr>
</table><p>
or
</p><table class="display dcenter"><tr class="c016"><td class="dcell">
</td><td class="dcell"><table class="display"><tr><td class="dcell c012">&nbsp;</td></tr>
<tr><td class="dcell c012">min</td></tr>
<tr><td class="dcell c012">α ∈ ℝ<sup><span class="c007">p</span></sup></td></tr>
</table></td><td class="dcell"> </td><td class="dcell"><table class="display"><tr><td class="dcell c012">1</td></tr>
<tr><td class="hbar"></td></tr>
<tr><td class="dcell c012">2</td></tr>
</table></td><td class="dcell">||diag(β)(<span class="c009">x</span>−<span class="c009">D</span>α)||<sub>2</sub><sup>2</sup> +λ||α||<sub>0</sub>.
    (7)</td></tr>
</table><p>
where diag(β) is a diagonal matrix with the entries of β on the diagonal.</p><table class="lstframe c011"><tr><td class="mouselstlisting"><span class="c001">% <br>
% Usage:   A=mexOMPMask(X,D,B,param);<br>
% or       [A path]=mexOMPMask(X,D,B,param);<br>
%<br>
% Name: mexOMPMask<br>
%<br>
% Description: mexOMPMask is a variant of mexOMP that allow using<br>
%     a binary mask B<br>
%     <br>
%     for all columns x of X, and columns beta of B, it computes a column <br>
%         alpha of A by addressing<br>
%         min_{alpha} ||alpha||_0  s.t  ||diag(beta)*(x-Dalpha)||_2^2 <br>
%                                                               &lt;= eps*||beta||_0/m<br>
%         or<br>
%         min_{alpha} ||diag(beta)*(x-Dalpha)||_2^2  s.t. ||alpha||_0 &lt;= L<br>
%         or<br>
%         min_{alpha} 0.5||diag(beta)*(x-Dalpha)||_2^2  + lambda||alpha||_0<br>
%         <br>
%<br>
% Inputs: X:  double m x n matrix   (input signals)<br>
%            m is the signal size<br>
%            n is the number of signals to decompose<br>
%         D:  double m x p matrix   (dictionary)<br>
%            p is the number of elements in the dictionary<br>
%            All the columns of D should have unit-norm !<br>
%         B:  boolean m x n matrix   (mask)<br>
%               p is the number of elements in the dictionary<br>
%         param: struct<br>
%            param.L (optional, maximum number of elements in each decomposition, <br>
%               min(m,p) by default)<br>
%            param.eps (optional, threshold on the squared l2-norm of the residual,<br>
%               0 by default<br>
%            param.lambda (optional, penalty parameter, 0 by default<br>
%            param.numThreads (optional, number of threads for exploiting<br>
%            multi-core / multi-cpus. By default, it takes the value -1,<br>
%            which automatically selects all the available CPUs/cores).<br>
%<br>
% Output: A: double sparse p x n matrix (output coefficients)<br>
%         path (optional): double dense p x L matrix <br>
%                                     (regularization path of the first signal)<br>
%<br>
% Note: this function admits a few experimental usages, which have not<br>
%     been extensively tested:<br>
%      - single precision setting (even though the output alpha is double <br>
%        precision)<br>
%      - Passing an int32 vector of length n to param.L provides<br>
%        a different parameter L for each input signal x_i<br>
%      - Passing a double vector of length n to param.eps and or param.lambda <br>
%        provides a different parameter eps (or lambda) for each input signal x_i<br>
%<br>
% Author: Julien Mairal, 2010</span></td></tr>
</table><p>
The following piece of code illustrates how to use this function.
</p><table class="lstframe c011"><tr><td class="mouselstlisting"><span class="c002">clear</span> <span class="c002">all</span>;<br>
<br>
<span class="c002">randn</span>(<span class="c003">'seed'</span>,0);<br>
<span class="c002">fprintf</span>(<span class="c003">'test mexOMPMask\n'</span>);<br>
<br>
<span class="c001">%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%<br>
% Decomposition of a large number of signals<br>
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%<br>
% Data are generated</span><br>
X=<span class="c002">randn</span>(100,100);<br>
X=X./repmat(<span class="c002">sqrt</span>(<span class="c002">sum</span>(X.^2)),[<span class="c002">size</span>(X,1) 1]);<br>
D=<span class="c002">randn</span>(100,20);<br>
D=D./repmat(<span class="c002">sqrt</span>(<span class="c002">sum</span>(D.^2)),[<span class="c002">size</span>(D,1) 1]);<br>
mask=(X &gt; 0); <span class="c001">% generating a binary mask<br>
<br>
% parameter of the optimization procedure are chosen</span><br>
param.L=20; <span class="c001">% not more than 20 non-zeros coefficients (default: min(size(D,1),size(D,2)))</span><br>
param.<span class="c002">eps</span>=0.01; <span class="c001">%</span><br>
param.numThreads=-1; <span class="c001">% number of processors/cores to use; the default choice is -1</span><br>
                     <span class="c001">% and uses all the cores of the machine</span><br>
<span class="c002">tic</span><br>
alpha=mexOMPMask(X,D,mask,param);<br>
t=<span class="c002">toc</span>;<br>
<span class="c002">toc<br>
<br>
fprintf</span>(<span class="c003">'%f signals processed per second\n'</span>,<span class="c002">size</span>(X,2)/t);</td></tr>
</table>
<!--TOC subsection id="sec14" Function mexRidgeRegression-->
<h3 class="subsection" id="sec14">4.3  Function mexRidgeRegression</h3><!--SEC END --><p>
This is a ridge regression solver using a conjugate gradient solver. 
</p><table class="lstframe c011"><tr><td class="mouselstlisting"><span class="c001">% <br>
% Usage: [W]=mexRidgeRegression(Y,X,W0,param);<br>
%<br>
% Name: mexRidgeRegression<br>
%<br>
% Description: mexFistaFlat solves sparse regularized problems.<br>
%         X is a design matrix of size m x p<br>
%         X=[x^1,...,x^n]', where the x_i's are the rows of X<br>
%         Y=[y^1,...,y^n] is a matrix of size m x n<br>
%         It implements a conjugate gradient solver for ridge regression<br>
%         <br>
% Inputs: Y:  double dense m x n matrix<br>
%         X:  double dense or sparse m x p matrix   <br>
%         W0:  double dense p x n matrix or p x Nn matrix (for multi-logistic loss)<br>
%              initial guess<br>
%         param: struct<br>
%            param.lambda (regularization parameter)<br>
%            param.numThreads (optional, number of threads for exploiting<br>
%                multi-core / multi-cpus. By default, it takes the value -1,<br>
%                which automatically selects all the available CPUs/cores).<br>
%            param.itermax (optional, maximum number of iterations, 100 by default)<br>
%            param.tol (optional, tolerance for stopping criteration, which is a relative duality gap<br>
%                if it is available, or a relative change of parameters).<br>
%<br>
% Output:  W:  double dense p x n matrix <br>
<br>
% Author: Julien Mairal, 2013</span></td></tr>
</table><p>
The following piece of code illustrates how to use this function.
</p><table class="lstframe c011"><tr><td class="mouselstlisting"><span class="c002">format</span> compact;<br>
<span class="c002">randn</span>(<span class="c003">'seed'</span>,0);<br>
param.numThreads=-1; <span class="c001">% all cores (-1 by default)</span><br>
param.lambda=0.05; <span class="c001">% regularization parameter</span><br>
<br>
X=<span class="c002">randn</span>(100,200);<br>
X=X-repmat(<span class="c002">mean</span>(X),[<span class="c002">size</span>(X,1) 1]);<br>
X=mexNormalize(X);<br>
Y=<span class="c002">randn</span>(100,1);<br>
Y=Y-repmat(<span class="c002">mean</span>(Y),[<span class="c002">size</span>(Y,1) 1]);<br>
Y=mexNormalize(Y);<br>
W0=<span class="c002">zeros</span>(<span class="c002">size</span>(X,2),<span class="c002">size</span>(Y,2));<br>
<span class="c001">% Regression experiments <br>
% 100 regression problems with the same design matrix X.</span><br>
<span class="c002">fprintf</span>(<span class="c003">'\nVarious regression experiments\n'</span>);<br>
<span class="c002">fprintf</span>(<span class="c003">'\nRidge Regression with conjugate gradient solver\n'</span>);<br>
<span class="c002">tic</span><br>
[W]=mexRidgeRegression(Y,X,W0,param);<br>
t=<span class="c002">toc</span></td></tr>
</table>
<!--TOC subsection id="sec15" Function mexLasso-->
<h3 class="subsection" id="sec15">4.4  Function mexLasso</h3><!--SEC END --><p>
This is a fast implementation of the LARS algorithm [<a href="#efron">9</a>] (variant for solving the Lasso) for solving the Lasso or Elastic-Net. Given a matrix of signals <span class="c009">X</span>=[<span class="c009">x</span><sup>1</sup>,…,<span class="c009">x</span><sup><span class="c007">n</span></sup>] in ℝ<sup><span class="c007">m</span> × <span class="c007">n</span></sup> and a dictionary <span class="c009">D</span> in ℝ<sup><span class="c007">m</span> × <span class="c007">p</span></sup>, depending on the input parameters, the algorithm returns a matrix of coefficients <span class="c009">A</span>=[α<sup>1</sup>,…,α<sup><span class="c007">n</span></sup>] in ℝ<sup><span class="c007">p</span> × <span class="c007">n</span></sup> such that for every column <span class="c009">x</span> of <span class="c009">X</span>, the corresponding column α of <span class="c009">A</span> is the solution of
</p><table class="display dcenter"><tr class="c016"><td class="dcell">
</td><td class="dcell"><table class="display"><tr><td class="dcell c012">&nbsp;</td></tr>
<tr><td class="dcell c012">min</td></tr>
<tr><td class="dcell c012">α ∈ ℝ<sup><span class="c007">p</span></sup></td></tr>
</table></td><td class="dcell"> ||<span class="c009">x</span>−<span class="c009">D</span>α||<sub>2</sub><sup>2</sup>   s.t.   ||α||<sub>1</sub> ≤ λ,
    (8)</td></tr>
</table><p>
or 
</p><table class="display dcenter"><tr class="c016"><td class="dcell">
</td><td class="dcell"><table class="display"><tr><td class="dcell c012">&nbsp;</td></tr>
<tr><td class="dcell c012">min</td></tr>
<tr><td class="dcell c012">α ∈ ℝ<sup><span class="c007">p</span></sup></td></tr>
</table></td><td class="dcell">  ||α||<sub>1</sub>   s.t.   ||<span class="c009">x</span>−<span class="c009">D</span>α||<sub>2</sub><sup>2</sup> ≤ λ, <a id="eq:lasso2"></a>
    (9)</td></tr>
</table><p>
or
</p><table class="display dcenter"><tr class="c016"><td class="dcell">
</td><td class="dcell"><table class="display"><tr><td class="dcell c012">&nbsp;</td></tr>
<tr><td class="dcell c012">min</td></tr>
<tr><td class="dcell c012">α ∈ ℝ<sup><span class="c007">p</span></sup></td></tr>
</table></td><td class="dcell"> </td><td class="dcell"><table class="display"><tr><td class="dcell c012">1</td></tr>
<tr><td class="hbar"></td></tr>
<tr><td class="dcell c012">2</td></tr>
</table></td><td class="dcell">||<span class="c009">x</span>−<span class="c009">D</span>α||<sub>2</sub><sup>2</sup> + λ ||α||<sub>1</sub> + </td><td class="dcell"><table class="display"><tr><td class="dcell c012">λ<sub>2</sub></td></tr>
<tr><td class="hbar"></td></tr>
<tr><td class="dcell c012">2</td></tr>
</table></td><td class="dcell">||α||<sub>2</sub><sup>2</sup>. <a id="eq:lasso"></a>
    (10)</td></tr>
</table><p>
For efficiency reasons, the method first compute the covariance matrix <span class="c009">D</span><sup><span class="c007">T</span></sup><span class="c009">D</span>, then
for each signal, it computes <span class="c009">D</span><sup><span class="c007">T</span></sup><span class="c009">x</span> and performs the decomposition with a
Cholesky-based algorithm (see [<a href="#efron">9</a>] for instance). The implementation
has also an option to add <span class="c010">positivity constraints</span> on the solutions
α. When the solution is very sparse and the problem size is
reasonable, this approach can be very efficient. Moreover, it gives the
solution with an exact precision, and its performance does not depend on the
correlation of the dictionary elements, except when the solution is not unique
(the algorithm breaks in this case).</p><p>Note that mexLasso can return the whole regularization path of the first signal <span class="c009">x</span><sub>1</sub> 
and can handle implicitely the matrix <span class="c009">D</span> if the quantities <span class="c009">D</span><sup><span class="c007">T</span></sup><span class="c009">D</span> and <span class="c009">D</span><sup><span class="c007">T</span></sup><span class="c009">x</span> are passed
as an argument, see below:</p><table class="lstframe c011"><tr><td class="mouselstlisting"><span class="c001">% <br>
% Usage:  [A [path]]=mexLasso(X,D,param);<br>
%  or:    [A [path]]=mexLasso(X,Q,q,param);<br>
%<br>
% Name: mexLasso<br>
%<br>
% Description: mexLasso is an efficient implementation of the<br>
%     homotopy-LARS algorithm for solving the Lasso. <br>
%     <br>
%     if the function is called this way [A [path]]=mexLasso(X,D,param),<br>
%     it aims at addressing the following problems<br>
%     for all columns x of X, it computes one column alpha of A<br>
%     that solves<br>
%       1) when param.mode=0<br>
%         min_{alpha} ||x-Dalpha||_2^2 s.t. ||alpha||_1 &lt;= lambda<br>
%       2) when param.mode=1<br>
%         min_{alpha} ||alpha||_1 s.t. ||x-Dalpha||_2^2 &lt;= lambda<br>
%       3) when param.mode=2<br>
%         min_{alpha} 0.5||x-Dalpha||_2^2 + lambda||alpha||_1 +0.5 lambda2||alpha||_2^2<br>
%<br>
%     if the function is called this way [A [path]]=mexLasso(X,Q,q,param),<br>
%     it solves the above optimisation problem, when Q=D'D and q=D'x.<br>
%<br>
%     Possibly, when param.pos=true, it solves the previous problems<br>
%     with positivity constraints on the vectors alpha<br>
%<br>
% Inputs: X:  double m x n matrix   (input signals)<br>
%               m is the signal size<br>
%               n is the number of signals to decompose<br>
%         D:  double m x p matrix   (dictionary)<br>
%               p is the number of elements in the dictionary<br>
%         param: struct<br>
%               param.lambda  (parameter)<br>
%               param.lambda2  (optional parameter for solving the Elastic-Net)<br>
%                              for mode=0 and mode=1, it adds a ridge on the Gram Matrix<br>
%               param.L (optional), maximum number of steps of the homotopy algorithm (can<br>
%                        be used as a stopping criterion)<br>
%               param.pos (optional, adds non-negativity constraints on the<br>
%                 coefficients, false by default)<br>
%               param.mode (see above, by default: 2)<br>
%               param.numThreads (optional, number of threads for exploiting<br>
%                 multi-core / multi-cpus. By default, it takes the value -1,<br>
%                 which automatically selects all the available CPUs/cores).<br>
%               param.cholesky (optional, default false),  choose between Cholesky <br>
%                 implementation or one based on the matrix inversion Lemma<br>
%               param.ols (optional, default false), perform an orthogonal projection<br>
%                 before returning the solution.<br>
%               param.max_length_path (optional) maximum length of the path, by default 4*p<br>
%<br>
% Output: A: double sparse p x n matrix (output coefficients)<br>
%         path: optional,  returns the regularisation path for the first signal<br>
%<br>
% Note: this function admits a few experimental usages, which have not<br>
%     been extensively tested:<br>
%         - single precision setting (even though the output alpha is double <br>
%           precision)<br>
%<br>
% Author: Julien Mairal, 2009</span></td></tr>
</table><p>
The following piece of code illustrates how to use this function.
</p><table class="lstframe c011"><tr><td class="mouselstlisting"><span class="c002">clear</span> <span class="c002">all</span>;<br>
<br>
<span class="c002">randn</span>(<span class="c003">'seed'</span>,0);<br>
<span class="c002">fprintf</span>(<span class="c003">'test mexLasso\n'</span>);<br>
<span class="c001">%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%<br>
% Decomposition of a large number of signals<br>
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%<br>
% Data are generated</span><br>
X=<span class="c002">randn</span>(100,100000);<br>
X=X./repmat(<span class="c002">sqrt</span>(<span class="c002">sum</span>(X.^2)),[<span class="c002">size</span>(X,1) 1]);<br>
D=<span class="c002">randn</span>(100,200);<br>
D=D./repmat(<span class="c002">sqrt</span>(<span class="c002">sum</span>(D.^2)),[<span class="c002">size</span>(D,1) 1]);<br>
<br>
<span class="c001">% parameter of the optimization procedure are chosen<br>
%param.L=20; % not more than 20 non-zeros coefficients (default: min(size(D,1),size(D,2)))</span><br>
param.lambda=0.15; <span class="c001">% not more than 20 non-zeros coefficients</span><br>
param.numThreads=-1; <span class="c001">% number of processors/cores to use; the default choice is -1</span><br>
                     <span class="c001">% and uses all the cores of the machine</span><br>
param.mode=2;        <span class="c001">% penalized formulation</span><br>
<br>
<span class="c002">tic</span><br>
alpha=mexLasso(X,D,param);<br>
t=<span class="c002">toc<br>
fprintf</span>(<span class="c003">'%f signals processed per second\n'</span>,<span class="c002">size</span>(X,2)/t);<br>
<br>
<span class="c001">%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%<br>
% Regularization path of a single signal <br>
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%</span><br>
X=<span class="c002">randn</span>(64,1);<br>
D=<span class="c002">randn</span>(64,10);<br>
D=D./repmat(<span class="c002">sqrt</span>(<span class="c002">sum</span>(D.^2)),[<span class="c002">size</span>(D,1) 1]);<br>
param.lambda=0;<br>
[alpha <span class="c002">path</span>]=mexLasso(X,D,param);</td></tr>
</table>
<!--TOC subsection id="sec16" Function mexLassoWeighted-->
<h3 class="subsection" id="sec16">4.5  Function mexLassoWeighted</h3><!--SEC END --><p>
This is a fast implementation of a weighted version of LARS [<a href="#efron">9</a>]. Given a matrix of signals <span class="c009">X</span>=[<span class="c009">x</span><sup>1</sup>,…,<span class="c009">x</span><sup><span class="c007">n</span></sup>] in ℝ<sup><span class="c007">m</span> × <span class="c007">n</span></sup>, a matrix of weights <span class="c009">W</span>=[<span class="c009">w</span><sup>1</sup>,…,<span class="c009">w</span><sup><span class="c007">n</span></sup>] ∈ ℝ<sup><span class="c007">p</span> × <span class="c007">n</span></sup>, and a dictionary <span class="c009">D</span> in ℝ<sup><span class="c007">m</span> × <span class="c007">p</span></sup>, depending on the input parameters, the algorithm returns a matrix of coefficients <span class="c009">A</span>=[α<sup>1</sup>,…,α<sup><span class="c007">n</span></sup>] in ℝ<sup><span class="c007">p</span> × <span class="c007">n</span></sup>,
such that for every column <span class="c009">x</span> of <span class="c009">X</span>, <span class="c009">w</span> of <span class="c009">W</span>, it computes a column α of <span class="c009">A</span>, which is the solution of
</p><table class="display dcenter"><tr class="c016"><td class="dcell">
</td><td class="dcell"><table class="display"><tr><td class="dcell c012">&nbsp;</td></tr>
<tr><td class="dcell c012">min</td></tr>
<tr><td class="dcell c012">α ∈ ℝ<sup><span class="c007">p</span></sup></td></tr>
</table></td><td class="dcell"> ||<span class="c009">x</span>−<span class="c009">D</span>α||<sub>2</sub><sup>2</sup>   s.t.   ||diag(<span class="c009">w</span>)α||<sub>1</sub> ≤ λ,
    (11)</td></tr>
</table><p>
or 
</p><table class="display dcenter"><tr class="c016"><td class="dcell">
</td><td class="dcell"><table class="display"><tr><td class="dcell c012">&nbsp;</td></tr>
<tr><td class="dcell c012">min</td></tr>
<tr><td class="dcell c012">α ∈ ℝ<sup><span class="c007">p</span></sup></td></tr>
</table></td><td class="dcell">  ||diag(<span class="c009">w</span>)α||<sub>1</sub>   s.t.   ||<span class="c009">x</span>−<span class="c009">D</span>α||<sub>2</sub><sup>2</sup> ≤ λ,
    (12)</td></tr>
</table><p>
or
</p><table class="display dcenter"><tr class="c016"><td class="dcell">
</td><td class="dcell"><table class="display"><tr><td class="dcell c012">&nbsp;</td></tr>
<tr><td class="dcell c012">min</td></tr>
<tr><td class="dcell c012">α ∈ ℝ<sup><span class="c007">p</span></sup></td></tr>
</table></td><td class="dcell"> </td><td class="dcell"><table class="display"><tr><td class="dcell c012">1</td></tr>
<tr><td class="hbar"></td></tr>
<tr><td class="dcell c012">2</td></tr>
</table></td><td class="dcell">||<span class="c009">x</span>−<span class="c009">D</span>α||<sub>2</sub><sup>2</sup> + λ ||diag(<span class="c009">w</span>)α||<sub>1</sub>.
    (13)</td></tr>
</table><p>
The implementation has also an option to add <span class="c010">positivity constraints</span> on
the solutions α. This function is potentially useful for
implementing efficiently the randomized Lasso of [<a href="#meinshausen">28</a>], or reweighted-ℓ<sub>1</sub> schemes [<a href="#candes4">4</a>].</p><table class="lstframe c011"><tr><td class="mouselstlisting"><span class="c001">% <br>
% Usage:  A=mexLassoWeighted(X,D,W,param);<br>
%<br>
% Name: mexLassoWeighted.  <br>
%<br>
% WARNING: This function has not been tested intensively<br>
%<br>
% Description: mexLassoWeighted is an efficient implementation of the<br>
%     LARS algorithm for solving the weighted Lasso. It is optimized<br>
%     for solving a large number of small or medium-sized <br>
%     decomposition problem (and not for a single large one).<br>
%     It first computes the Gram matrix D'D and then perform<br>
%     a Cholesky-based OMP of the input signals in parallel.<br>
%     For all columns x of X, and w of W, it computes one column alpha of A<br>
%     which is the solution of<br>
%       1) when param.mode=0<br>
%         min_{alpha} ||x-Dalpha||_2^2   s.t.  <br>
%                                     ||diag(w)alpha||_1 &lt;= lambda<br>
%       2) when param.mode=1<br>
%         min_{alpha} ||diag(w)alpha||_1  s.t.<br>
%                                        ||x-Dalpha||_2^2 &lt;= lambda<br>
%       3) when param.mode=2<br>
%         min_{alpha} 0.5||x-Dalpha||_2^2  +  <br>
%                                         lambda||diag(w)alpha||_1 <br>
%     Possibly, when param.pos=true, it solves the previous problems<br>
%     with positivity constraints on the vectors alpha<br>
%<br>
% Inputs: X:  double m x n matrix   (input signals)<br>
%               m is the signal size<br>
%               n is the number of signals to decompose<br>
%         D:  double m x p matrix   (dictionary)<br>
%               p is the number of elements in the dictionary<br>
%         W:  double p x n matrix   (weights)<br>
%         param: struct<br>
%            param.lambda  (parameter)<br>
%            param.L (optional, maximum number of elements of each <br>
%            decomposition)<br>
%            param.pos (optional, adds positivity constraints on the<br>
%            coefficients, false by default)<br>
%            param.mode (see above, by default: 2)<br>
%            param.numThreads (optional, number of threads for exploiting<br>
%            multi-core / multi-cpus. By default, it takes the value -1,<br>
%            which automatically selects all the available CPUs/cores).<br>
%<br>
% Output:  A: double sparse p x n matrix (output coefficients)<br>
%<br>
% Note: this function admits a few experimental usages, which have not<br>
%     been extensively tested:<br>
%         - single precision setting (even though the output alpha is double <br>
%           precision)<br>
%<br>
% Author: Julien Mairal, 2009</span></td></tr>
</table><p>
The following piece of code illustrates how to use this function.
</p><table class="lstframe c011"><tr><td class="mouselstlisting"><span class="c002">clear</span> <span class="c002">all</span>;<br>
<br>
<span class="c002">fprintf</span>(<span class="c003">'test Lasso weighted\n'</span>);<br>
<span class="c002">randn</span>(<span class="c003">'seed'</span>,0);<br>
<span class="c001">% Data are generated</span><br>
X=<span class="c002">randn</span>(64,10000);<br>
X=X./repmat(<span class="c002">sqrt</span>(<span class="c002">sum</span>(X.^2)),[<span class="c002">size</span>(X,1) 1]);<br>
D=<span class="c002">randn</span>(64,256);<br>
D=D./repmat(<span class="c002">sqrt</span>(<span class="c002">sum</span>(D.^2)),[<span class="c002">size</span>(D,1) 1]);<br>
<br>
<span class="c001">% parameter of the optimization procedure are chosen</span><br>
param.L=20; <span class="c001">% not more than 20 non-zeros coefficients (default: min(size(D,1),size(D,2)))</span><br>
param.lambda=0.15; <span class="c001">% not more than 20 non-zeros coefficients</span><br>
param.numThreads=8; <span class="c001">% number of processors/cores to use; the default choice is -1</span><br>
                    <span class="c001">% and uses all the cores of the machine</span><br>
param.mode=2;       <span class="c001">% penalized formulation</span><br>
<br>
W=<span class="c002">rand</span>(<span class="c002">size</span>(D,2),<span class="c002">size</span>(X,2));<br>
<br>
<span class="c002">tic</span><br>
alpha=mexLassoWeighted(X,D,W,param);<br>
t=<span class="c002">toc</span>;<br>
<span class="c002">toc<br>
<br>
fprintf</span>(<span class="c003">'%f signals processed per second\n'</span>,<span class="c002">size</span>(X,2)/t);</td></tr>
</table>
<!--TOC subsection id="sec17" Function mexLassoMask-->
<h3 class="subsection" id="sec17">4.6  Function mexLassoMask</h3><!--SEC END --><p>
This is a variant of mexLasso with the possibility of adding a mask <span class="c009">B</span>=[β<sup>1</sup>,…,β<sup><span class="c007">n</span></sup>], as in mexOMPMask. 
For every column <span class="c009">x</span> of <span class="c009">X</span>, β of <span class="c009">B</span>, it computes a column α of <span class="c009">A</span>, which is the solution of
</p><table class="display dcenter"><tr class="c016"><td class="dcell">
</td><td class="dcell"><table class="display"><tr><td class="dcell c012">&nbsp;</td></tr>
<tr><td class="dcell c012">min</td></tr>
<tr><td class="dcell c012">α ∈ ℝ<sup><span class="c007">p</span></sup></td></tr>
</table></td><td class="dcell"> ||diag(β)(<span class="c009">x</span>−<span class="c009">D</span>α)||<sub>2</sub><sup>2</sup>   s.t.   ||α||<sub>1</sub> ≤ λ,
    (14)</td></tr>
</table><p>
or 
</p><table class="display dcenter"><tr class="c016"><td class="dcell">
</td><td class="dcell"><table class="display"><tr><td class="dcell c012">&nbsp;</td></tr>
<tr><td class="dcell c012">min</td></tr>
<tr><td class="dcell c012">α ∈ ℝ<sup><span class="c007">p</span></sup></td></tr>
</table></td><td class="dcell">  ||α||<sub>1</sub>   s.t.   ||diag(β)(<span class="c009">x</span>−<span class="c009">D</span>α)||<sub>2</sub><sup>2</sup> ≤ λ</td><td class="dcell"><table class="display"><tr><td class="dcell c012">||β||<sub>0</sub></td></tr>
<tr><td class="hbar"></td></tr>
<tr><td class="dcell c012"><span class="c007">m</span></td></tr>
</table></td><td class="dcell">, 
    (15)</td></tr>
</table><p>
or
</p><table class="display dcenter"><tr class="c016"><td class="dcell">
</td><td class="dcell"><table class="display"><tr><td class="dcell c012">&nbsp;</td></tr>
<tr><td class="dcell c012">min</td></tr>
<tr><td class="dcell c012">α ∈ ℝ<sup><span class="c007">p</span></sup></td></tr>
</table></td><td class="dcell"> </td><td class="dcell"><table class="display"><tr><td class="dcell c012">1</td></tr>
<tr><td class="hbar"></td></tr>
<tr><td class="dcell c012">2</td></tr>
</table></td><td class="dcell">||diag(β)(<span class="c009">x</span>−<span class="c009">D</span>α)||<sub>2</sub><sup>2</sup> + λ </td><td class="dcell"><table class="display"><tr><td class="dcell c012">||β||<sub>0</sub></td></tr>
<tr><td class="hbar"></td></tr>
<tr><td class="dcell c012"><span class="c007">m</span></td></tr>
</table></td><td class="dcell">||α||<sub>1</sub> + </td><td class="dcell"><table class="display"><tr><td class="dcell c012">λ<sub>2</sub></td></tr>
<tr><td class="hbar"></td></tr>
<tr><td class="dcell c012">2</td></tr>
</table></td><td class="dcell">||α||<sub>2</sub><sup>2</sup>. 
    (16)</td></tr>
</table><table class="lstframe c011"><tr><td class="mouselstlisting"><span class="c001">% <br>
% Usage:   A=mexLassoMask(X,D,B,param);<br>
%<br>
% Name: mexLassoMask<br>
%<br>
% Description: mexLasso is a variant of mexLasso that handles<br>
%     binary masks. It aims at addressing the following problems<br>
%     for all columns x of X, and beta of B, it computes one column alpha of A<br>
%     that solves<br>
%       1) when param.mode=0<br>
%         min_{alpha} ||diag(beta)(x-Dalpha)||_2^2 s.t. ||alpha||_1 &lt;= lambda<br>
%       2) when param.mode=1<br>
%         min_{alpha} ||alpha||_1 s.t. ||diag(beta)(x-Dalpha)||_2^2 <br>
%                                                              &lt;= lambda*||beta||_0/m<br>
%       3) when param.mode=2<br>
%         min_{alpha} 0.5||diag(beta)(x-Dalpha)||_2^2 +<br>
%                                                 lambda*(||beta||_0/m)*||alpha||_1 +<br>
%                                                 (lambda2/2)||alpha||_2^2<br>
%     Possibly, when param.pos=true, it solves the previous problems<br>
%     with positivity constraints on the vectors alpha<br>
%<br>
% Inputs: X:  double m x n matrix   (input signals)<br>
%               m is the signal size<br>
%               n is the number of signals to decompose<br>
%         D:  double m x p matrix   (dictionary)<br>
%               p is the number of elements in the dictionary<br>
%         B:  boolean m x n matrix   (mask)<br>
%               p is the number of elements in the dictionary<br>
%         param: struct<br>
%               param.lambda  (parameter)<br>
%               param.L (optional, maximum number of elements of each <br>
%                 decomposition)<br>
%               param.pos (optional, adds positivity constraints on the<br>
%                 coefficients, false by default)<br>
%               param.mode (see above, by default: 2)<br>
%               param.lambda2  (optional parameter for solving the Elastic-Net)<br>
%                              for mode=0 and mode=1, it adds a ridge on the Gram Matrix<br>
%               param.numThreads (optional, number of threads for exploiting<br>
%                 multi-core / multi-cpus. By default, it takes the value -1,<br>
%                 which automatically selects all the available CPUs/cores).<br>
%<br>
% Output: A: double sparse p x n matrix (output coefficients)<br>
%<br>
% Note: this function admits a few experimental usages, which have not<br>
%     been extensively tested:<br>
%         - single precision setting (even though the output alpha is double <br>
%           precision)<br>
%<br>
% Author: Julien Mairal, 2010</span></td></tr>
</table><p>
The following piece of code illustrates how to use this function.
</p><table class="lstframe c011"><tr><td class="mouselstlisting"><span class="c002">clear</span> <span class="c002">all</span>;<br>
<br>
<span class="c002">randn</span>(<span class="c003">'seed'</span>,0);<br>
<span class="c002">fprintf</span>(<span class="c003">'test mexLasso\n'</span>);<br>
<br>
<span class="c001">%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%<br>
% Decomposition of a large number of signals<br>
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%<br>
% Data are generated</span><br>
X=<span class="c002">randn</span>(100,100);<br>
X=X./repmat(<span class="c002">sqrt</span>(<span class="c002">sum</span>(X.^2)),[<span class="c002">size</span>(X,1) 1]);<br>
D=<span class="c002">randn</span>(100,20);<br>
D=D./repmat(<span class="c002">sqrt</span>(<span class="c002">sum</span>(D.^2)),[<span class="c002">size</span>(D,1) 1]);<br>
mask=(X &gt; 0); <span class="c001">% generating a binary mask<br>
<br>
% parameter of the optimization procedure are chosen<br>
%param.L=20; % not more than 20 non-zeros coefficients (default: min(size(D,1),size(D,2)))</span><br>
param.lambda=0.15; <span class="c001">% not more than 20 non-zeros coefficients</span><br>
param.numThreads=-1; <span class="c001">% number of processors/cores to use; the default choice is -1</span><br>
                     <span class="c001">% and uses all the cores of the machine</span><br>
param.mode=2;        <span class="c001">% penalized formulation</span><br>
<br>
<span class="c002">tic</span><br>
alpha=mexLassoMask(X,D,mask,param);<br>
t=<span class="c002">toc</span>;<br>
<span class="c002">toc<br>
<br>
fprintf</span>(<span class="c003">'%f signals processed per second\n'</span>,<span class="c002">size</span>(X,2)/t);</td></tr>
</table>
<!--TOC subsection id="sec18" Function mexCD-->
<h3 class="subsection" id="sec18">4.7  Function mexCD</h3><!--SEC END --><p>
Coordinate-descent approach for solving Eq. (<a href="#eq%3Alasso">10</a>) and
Eq. (<a href="#eq%3Alasso2">9</a>). Note that unlike mexLasso, it is not implemented to solve the Elastic-Net formulation.
To solve Eq. (<a href="#eq%3Alasso2">9</a>), the algorithm solves a
sequence of problems of the form (<a href="#eq%3Alasso">10</a>) using simple heuristics.
Coordinate descent is very simple and in practice very powerful. It performs
better when the correlation between the dictionary elements is small. </p><table class="lstframe c011"><tr><td class="mouselstlisting"><span class="c001">% <br>
% Usage:   A=mexCD(X,D,A0,param);<br>
%<br>
% Name: mexCD<br>
%<br>
% Description: mexCD addresses l1-decomposition problem with a <br>
%     coordinate descent type of approach.<br>
%     It is optimized for solving a large number of small or medium-sized <br>
%     decomposition problem (and not for a single large one).<br>
%     It first computes the Gram matrix D'D.<br>
%     This method is particularly well adapted when there is low <br>
%     correlation between the dictionary elements and when one can benefit <br>
%     from a warm restart.<br>
%     It aims at addressing the two following problems<br>
%     for all columns x of X, it computes a column alpha of A such that<br>
%       2) when param.mode=1<br>
%         min_{alpha} ||alpha||_1 s.t. ||x-Dalpha||_2^2 &lt;= lambda<br>
%         For this constraint setting, the method solves a sequence of <br>
%         penalized problems (corresponding to param.mode=2) and looks<br>
%         for the corresponding Lagrange multplier with a simple but<br>
%         efficient heuristic.<br>
%       3) when param.mode=2<br>
%         min_{alpha} 0.5||x-Dalpha||_2^2 + lambda||alpha||_1 <br>
%<br>
% Inputs: X:  double m x n matrix   (input signals)<br>
%               m is the signal size<br>
%               n is the number of signals to decompose<br>
%         D:  double m x p matrix   (dictionary)<br>
%               p is the number of elements in the dictionary<br>
%               All the columns of D should have unit-norm !<br>
%         A0:  double sparse p x n matrix   (initial guess)<br>
%         param: struct<br>
%            param.lambda  (parameter)<br>
%            param.mode (optional, see above, by default 2)<br>
%            param.itermax (maximum number of iterations)<br>
%            param.tol (tolerance parameter)<br>
%            param.numThreads (optional, number of threads for exploiting<br>
%            multi-core / multi-cpus. By default, it takes the value -1,<br>
%            which automatically selects all the available CPUs/cores).<br>
%<br>
% Output: A: double sparse p x n matrix (output coefficients)<br>
%<br>
% Note: this function admits a few experimental usages, which have not<br>
%     been extensively tested:<br>
%         - single precision setting (even though the output alpha <br>
%           is double precision)<br>
%<br>
% Author: Julien Mairal, 2009</span></td></tr>
</table><p>
The following piece of code illustrates how to use this function.
</p><table class="lstframe c011"><tr><td class="mouselstlisting"><span class="c002">clear</span> <span class="c002">all</span>;<br>
<br>
<span class="c002">fprintf</span>(<span class="c003">'test mexCD\n'</span>);<br>
<span class="c002">randn</span>(<span class="c003">'seed'</span>,0);<br>
<span class="c001">% Data are generated</span><br>
X=<span class="c002">randn</span>(64,100);<br>
X=X./repmat(<span class="c002">sqrt</span>(<span class="c002">sum</span>(X.^2)),[<span class="c002">size</span>(X,1) 1]);<br>
D=<span class="c002">randn</span>(64,100);<br>
D=D./repmat(<span class="c002">sqrt</span>(<span class="c002">sum</span>(D.^2)),[<span class="c002">size</span>(D,1) 1]);<br>
<br>
<span class="c001">% parameter of the optimization procedure are chosen</span><br>
param.lambda=0.015; <span class="c001">% not more than 20 non-zeros coefficients</span><br>
param.numThreads=4; <span class="c001">% number of processors/cores to use; the default choice is -1</span><br>
param.mode=2;       <span class="c001">% penalized formulation</span><br>
<br>
<span class="c002">tic</span><br>
alpha=mexLasso(X,D,param);<br>
t=<span class="c002">toc</span>;<br>
<span class="c002">toc</span><br>
E=<span class="c002">mean</span>(0.5*<span class="c002">sum</span>((X-D*alpha).^2)+param.lambda*<span class="c002">sum</span>(<span class="c002">abs</span>(alpha)));<br>
<span class="c002">fprintf</span>(<span class="c003">'%f signals processed per second for LARS\n'</span>,<span class="c002">size</span>(X,2)/t);<br>
<span class="c002">fprintf</span>(<span class="c003">'Objective function for LARS: %g\n'</span>,E);<br>
<br>
param.tol=0.001;<br>
param.itermax=1000;<br>
<span class="c002">tic</span><br>
alpha2=mexCD(X,D,<span class="c002">sparse</span>(<span class="c002">size</span>(alpha,1),<span class="c002">size</span>(alpha,2)),param);<br>
t=<span class="c002">toc</span>;<br>
<span class="c002">toc<br>
<br>
fprintf</span>(<span class="c003">'%f signals processed per second for CD\n'</span>,<span class="c002">size</span>(X,2)/t);<br>
E=<span class="c002">mean</span>(0.5*<span class="c002">sum</span>((X-D*alpha2).^2)+param.lambda*<span class="c002">sum</span>(<span class="c002">abs</span>(alpha2)));<br>
<span class="c002">fprintf</span>(<span class="c003">'Objective function for CD: %g\n'</span>,E);<br>
<span class="c002">fprintf</span>(<span class="c003">'With Random Design, CD can be much faster than LARS\n'</span>);</td></tr>
</table>
<!--TOC subsection id="sec19" Function mexSOMP-->
<h3 class="subsection" id="sec19">4.8  Function mexSOMP</h3><!--SEC END --><p>
This is a fast implementation of the Simultaneous Orthogonal Matching Pursuit algorithm. Given a set of matrices <span class="c009">X</span>=[<span class="c009">X</span><sup>1</sup>,…,<span class="c009">X</span><sup><span class="c007">n</span></sup>] in ℝ<sup><span class="c007">m</span> × <span class="c007">N</span></sup>, where the <span class="c009">X</span><sup><span class="c007">i</span></sup>’s are in ℝ<sup><span class="c007">m</span> × <span class="c007">n</span><sub><span class="c007">i</span></sub></sup>, and a dictionary <span class="c009">D</span> in ℝ<sup><span class="c007">m</span> × <span class="c007">p</span></sup>, the algorithm returns a matrix of coefficients <span class="c009">A</span>=[<span class="c009">A</span><sup>1</sup>,…,<span class="c009">A</span><sup><span class="c007">n</span></sup>] in ℝ<sup><span class="c007">p</span> × <span class="c007">N</span></sup> which is an approximate solution of the following NP-hard problem
</p><table class="display dcenter"><tr class="c016"><td class="dcell">
∀ <span class="c007">i</span>   </td><td class="dcell"><table class="display"><tr><td class="dcell c012">&nbsp;</td></tr>
<tr><td class="dcell c012">min</td></tr>
<tr><td class="dcell c012"><span class="c009">A</span><sup><span class="c007">i</span></sup> ∈ ℝ<sup><span class="c007">p</span> × <span class="c007">n</span><sub><span class="c007">i</span></sub></sup></td></tr>
</table></td><td class="dcell"> ||<span class="c009">X</span><sup><span class="c007">i</span></sup>−<span class="c009">DA</span><sup><span class="c007">i</span></sup>||<sub><span class="c007">F</span></sub><sup>2</sup>   s.t.   ||<span class="c009">A</span><sup><span class="c007">i</span></sup>||<sub>0,∞</sub> ≤ <span class="c007">L</span>.
    (17)</td></tr>
</table><p>
or 
</p><table class="display dcenter"><tr class="c016"><td class="dcell">
∀ <span class="c007">i</span>   </td><td class="dcell"><table class="display"><tr><td class="dcell c012">&nbsp;</td></tr>
<tr><td class="dcell c012">min</td></tr>
<tr><td class="dcell c012"><span class="c009">A</span><sup><span class="c007">i</span></sup> ∈ ℝ<sup><span class="c007">p</span> × <span class="c007">n</span><sub><span class="c007">i</span></sub></sup></td></tr>
</table></td><td class="dcell">  ||<span class="c009">A</span><sup><span class="c007">i</span></sup>||<sub>0,∞</sub>   s.t.   ||<span class="c009">X</span><sup><span class="c007">i</span></sup>−<span class="c009">DA</span><sup><span class="c007">i</span></sup>||<sub><span class="c007">F</span></sub><sup>2</sup> ≤ ε <span class="c007">n</span><sub><span class="c007">i</span></sub>.
    (18)</td></tr>
</table><p>
To be efficient, the method first compute the covariance matrix <span class="c009">D</span><sup><span class="c007">T</span></sup><span class="c009">D</span>, then for each signal, it computes <span class="c009">D</span><sup><span class="c007">T</span></sup><span class="c009">X</span><sup><span class="c007">i</span></sup> and performs the decomposition with a Cholesky-based algorithm.</p><table class="lstframe c011"><tr><td class="mouselstlisting"><span class="c001">% <br>
% Usage:   alpha=mexSOMP(X,D,list_groups,param);<br>
%<br>
% Name: mexSOMP<br>
%     (this function has not been intensively tested).<br>
%<br>
% Description: mexSOMP is an efficient implementation of a<br>
%     Simultaneous Orthogonal Matching Pursuit algorithm. It is optimized<br>
%     for solving a large number of small or medium-sized <br>
%     decomposition problem (and not for a single large one).<br>
%     It first computes the Gram matrix D'D and then perform<br>
%     a Cholesky-based OMP of the input signals in parallel.<br>
%     It aims at addressing the following NP-hard problem<br>
%<br>
%     X is a matrix structured in groups of signals, which we denote<br>
%     by X=[X_1,...,X_n]<br>
%     <br>
%     for all matrices X_i of X, <br>
%         min_{A_i} ||A_i||_{0,infty}  s.t  ||X_i-D A_i||_2^2 &lt;= eps*n_i<br>
%         where n_i is the number of columns of X_i<br>
%<br>
%         or<br>
%<br>
%         min_{A_i} ||X_i-D A_i||_2^2  s.t. ||A_i||_{0,infty} &lt;= L<br>
%         <br>
%<br>
% Inputs: X:  double m x N matrix   (input signals)<br>
%            m is the signal size<br>
%            N is the total number of signals <br>
%         D:  double m x p matrix   (dictionary)<br>
%            p is the number of elements in the dictionary<br>
%            All the columns of D should have unit-norm !<br>
%         list_groups : int32 vector containing the indices (starting at 0)<br>
%            of the first elements of each groups.<br>
%         param: struct<br>
%            param.L (maximum number of elements in each decomposition)<br>
%            param.eps (threshold on the squared l2-norm of the residual<br>
%            param.numThreads (optional, number of threads for exploiting<br>
%            multi-core / multi-cpus. By default, it takes the value -1,<br>
%            which automatically selects all the available CPUs/cores).<br>
%<br>
% Output: alpha: double sparse p x N matrix (output coefficients)<br>
%<br>
% Note: this function admits a few experimental usages, which have not<br>
%     been extensively tested:<br>
%      - single precision setting (even though the output alpha is double <br>
%        precision)<br>
%<br>
% Author: Julien Mairal, 2010</span></td></tr>
</table><p>
The following piece of code illustrates how to use this function.
</p><table class="lstframe c011"><tr><td class="mouselstlisting"><span class="c002">clear</span> <span class="c002">all</span>;<br>
<span class="c002">randn</span>(<span class="c003">'seed'</span>,0);<br>
<br>
<span class="c002">fprintf</span>(<span class="c003">'test mexSOMP\n'</span>);<br>
<br>
<span class="c001">%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%<br>
% Decomposition of a large number of groups <br>
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%</span><br>
X=<span class="c002">randn</span>(64,10000);<br>
D=<span class="c002">randn</span>(64,200);<br>
D=D./repmat(<span class="c002">sqrt</span>(<span class="c002">sum</span>(D.^2)),[<span class="c002">size</span>(D,1) 1]);<br>
ind_groups=int32(0:10:9999); <span class="c001">% indices of the first signals in each group<br>
<br>
% parameter of the optimization procedure are chosen</span><br>
param.L=10; <span class="c001">% not more than 10 non-zeros coefficients</span><br>
param.<span class="c002">eps</span>=0.1; <span class="c001">% squared norm of the residual should be less than 0.1</span><br>
param.numThreads=-1; <span class="c001">% number of processors/cores to use; the default choice is -1</span><br>
                    <span class="c001">% and uses all the cores of the machine</span><br>
<span class="c002">tic</span><br>
alpha=mexSOMP(X,D,ind_groups,param);<br>
t=<span class="c002">toc<br>
fprintf</span>(<span class="c003">'%f signals processed per second\n'</span>,<span class="c002">size</span>(X,2)/t);</td></tr>
</table>
<!--TOC subsection id="sec20" Function mexL1L2BCD-->
<h3 class="subsection" id="sec20">4.9  Function mexL1L2BCD</h3><!--SEC END --><p>
This is a fast implementation of a simultaneous signal decomposition formulation. Given a set of matrices <span class="c009">X</span>=[<span class="c009">X</span><sup>1</sup>,…,<span class="c009">X</span><sup><span class="c007">n</span></sup>] in ℝ<sup><span class="c007">m</span> × <span class="c007">N</span></sup>, where the <span class="c009">X</span><sup><span class="c007">i</span></sup>’s are in ℝ<sup><span class="c007">m</span> × <span class="c007">n</span><sub><span class="c007">i</span></sub></sup>, and a dictionary <span class="c009">D</span> in ℝ<sup><span class="c007">m</span> × <span class="c007">p</span></sup>, the algorithm returns a matrix of coefficients <span class="c009">A</span>=[<span class="c009">A</span><sup>1</sup>,…,<span class="c009">A</span><sup><span class="c007">n</span></sup>] in ℝ<sup><span class="c007">p</span> × <span class="c007">N</span></sup> which is an approximate solution of the following NP-hard problem
</p><table class="display dcenter"><tr class="c016"><td class="dcell">
∀ <span class="c007">i</span>   </td><td class="dcell"><table class="display"><tr><td class="dcell c012">&nbsp;</td></tr>
<tr><td class="dcell c012">min</td></tr>
<tr><td class="dcell c012"><span class="c009">A</span><sup><span class="c007">i</span></sup> ∈ ℝ<sup><span class="c007">p</span> × <span class="c007">n</span><sub><span class="c007">i</span></sub></sup></td></tr>
</table></td><td class="dcell"> ||<span class="c009">X</span><sup><span class="c007">i</span></sup>−<span class="c009">DA</span><sup><span class="c007">i</span></sup>||<sub><span class="c007">F</span></sub><sup>2</sup>   s.t.   ||<span class="c009">A</span><sup><span class="c007">i</span></sup>||<sub>1,2</sub> ≤ </td><td class="dcell"><table class="display"><tr><td class="dcell c012">λ</td></tr>
<tr><td class="hbar"></td></tr>
<tr><td class="dcell c012"><span class="c007">n</span><sub><span class="c007">i</span></sub></td></tr>
</table></td><td class="dcell">.
    (19)</td></tr>
</table><p>
or 
</p><table class="display dcenter"><tr class="c016"><td class="dcell">
∀ <span class="c007">i</span>   </td><td class="dcell"><table class="display"><tr><td class="dcell c012">&nbsp;</td></tr>
<tr><td class="dcell c012">min</td></tr>
<tr><td class="dcell c012"><span class="c009">A</span><sup><span class="c007">i</span></sup> ∈ ℝ<sup><span class="c007">p</span> × <span class="c007">n</span><sub><span class="c007">i</span></sub></sup></td></tr>
</table></td><td class="dcell">  ||<span class="c009">A</span><sup><span class="c007">i</span></sup>||<sub>1,2</sub>   s.t.   ||<span class="c009">X</span><sup><span class="c007">i</span></sup>−<span class="c009">DA</span><sup><span class="c007">i</span></sup>||<sub><span class="c007">F</span></sub><sup>2</sup> ≤ λ <span class="c007">n</span><sub><span class="c007">i</span></sub>.
    (20)</td></tr>
</table><p>
To be efficient, the method first compute the covariance matrix <span class="c009">D</span><sup><span class="c007">T</span></sup><span class="c009">D</span>, then for each signal, it computes <span class="c009">D</span><sup><span class="c007">T</span></sup><span class="c009">X</span><sup><span class="c007">i</span></sup> and performs the decomposition with a Cholesky-based algorithm.</p><table class="lstframe c011"><tr><td class="mouselstlisting"><span class="c001">% <br>
% Usage:   alpha=mexL1L2BCD(X,D,alpha0,list_groups,param);<br>
%<br>
% Name: mexL1L2BCD<br>
%     (this function has not been intensively tested).<br>
%<br>
% Description: mexL1L2BCD is a solver for a <br>
%     Simultaneous signal decomposition formulation based on block <br>
%     coordinate descent.<br>
%<br>
%     X is a matrix structured in groups of signals, which we denote<br>
%     by X=[X_1,...,X_n]<br>
%     <br>
%     if param.mode=2, it solves<br>
%         for all matrices X_i of X, <br>
%         min_{A_i} 0.5||X_i-D A_i||_2^2 + lambda/sqrt(n_i)||A_i||_{1,2}  <br>
%         where n_i is the number of columns of X_i<br>
%     if param.mode=1, it solves<br>
%         min_{A_i} ||A_i||_{1,2} s.t. ||X_i-D A_i||_2^2  &lt;= n_i lambda<br>
%         <br>
%<br>
% Inputs: X:  double m x N matrix   (input signals)<br>
%            m is the signal size<br>
%            N is the total number of signals <br>
%         D:  double m x p matrix   (dictionary)<br>
%            p is the number of elements in the dictionary<br>
%         alpha0: double dense p x N matrix (initial solution)<br>
%         list_groups : int32 vector containing the indices (starting at 0)<br>
%            of the first elements of each groups.<br>
%         param: struct<br>
%            param.lambda (regularization parameter)<br>
%            param.mode (see above, by default 2)<br>
%            param.itermax (maximum number of iterations, by default 100)<br>
%            param.tol (tolerance parameter, by default 0.001)<br>
%            param.numThreads (optional, number of threads for exploiting<br>
%            multi-core / multi-cpus. By default, it takes the value -1,<br>
%            which automatically selects all the available CPUs/cores).<br>
%<br>
% Output: alpha: double sparse p x N matrix (output coefficients)<br>
%<br>
% Note: this function admits a few experimental usages, which have not<br>
%     been extensively tested:<br>
%      - single precision setting (even though the output alpha is double <br>
%        precision)<br>
%<br>
% Author: Julien Mairal, 2010</span></td></tr>
</table><p>
The following piece of code illustrates how to use this function.
</p><table class="lstframe c011"><tr><td class="mouselstlisting"><span class="c002">clear</span> <span class="c002">all</span>;<br>
<span class="c002">randn</span>(<span class="c003">'seed'</span>,0);<br>
<br>
<span class="c002">fprintf</span>(<span class="c003">'test mexL1L2BCD\n'</span>);<br>
<span class="c001">%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%<br>
% Decomposition of a large number of groups <br>
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%</span><br>
X=<span class="c002">randn</span>(64,100);<br>
D=<span class="c002">randn</span>(64,200);<br>
D=D./repmat(<span class="c002">sqrt</span>(<span class="c002">sum</span>(D.^2)),[<span class="c002">size</span>(D,1) 1]);<br>
ind_groups=int32(0:10:<span class="c002">size</span>(X,2)-1); <span class="c001">% indices of the first signals in each group<br>
<br>
% parameter of the optimization procedure are chosen</span><br>
param.itermax=100;<br>
param.tol=1e-3;<br>
param.mode=2; <span class="c001">% penalty mode</span><br>
param.lambda=0.15; <span class="c001">% squared norm of the residual should be less than 0.1</span><br>
param.numThreads=-1; <span class="c001">% number of processors/cores to use; the default choice is -1</span><br>
                    <span class="c001">% and uses all the cores of the machine</span><br>
<span class="c002">tic</span><br>
alpha0=<span class="c002">zeros</span>(<span class="c002">size</span>(D,2),<span class="c002">size</span>(X,2));<br>
alpha=mexL1L2BCD(X,D,alpha0,ind_groups,param);<br>
t=<span class="c002">toc<br>
fprintf</span>(<span class="c003">'%f signals processed per second\n'</span>,<span class="c002">size</span>(X,2)/t);</td></tr>
</table>
<!--TOC subsection id="sec21" Function mexSparseProject-->
<h3 class="subsection" id="sec21">4.10  Function mexSparseProject</h3><!--SEC END --><p>
This is a multi-purpose function, implementing fast algorithms for projecting
on convex sets, but it also solves the fused lasso signal approximation
problem. The proposed method is detailed in [<a href="#mairal9">21</a>]. The main problems
addressed by this function are the following: Given a matrix
<span class="c009">U</span>=[<span class="c009">u</span><sub>1</sub>,…,<span class="c009">u</span><sub><span class="c007">n</span></sub>] in ℝ<sup><span class="c007">m</span> × <span class="c007">n</span></sup>, it finds a matrix
<span class="c009">V</span>=[<span class="c009">v</span><sub>1</sub>,…,<span class="c009">v</span><sub><span class="c007">n</span></sub>] in ℝ<sup><span class="c007">m</span> × <span class="c007">n</span></sup> so that for all column <span class="c009">u</span> of <span class="c009">U</span>,
it computes a column <span class="c009">v</span> of <span class="c009">V</span> solving
</p><table class="display dcenter"><tr class="c016"><td class="dcell">
</td><td class="dcell"><table class="display"><tr><td class="dcell c012">&nbsp;</td></tr>
<tr><td class="dcell c012">min</td></tr>
<tr><td class="dcell c012"><span class="c009">v</span> ∈ ℝ<sup><span class="c007">m</span></sup></td></tr>
</table></td><td class="dcell"> ||<span class="c009">u</span>−<span class="c009">v</span>||<sub>2</sub><sup>2</sup>    s.t.   ||<span class="c009">v</span>||<sub>1</sub> ≤ τ,
    (21)</td></tr>
</table><p>
or
</p><table class="display dcenter"><tr class="c016"><td class="dcell">
</td><td class="dcell"><table class="display"><tr><td class="dcell c012">&nbsp;</td></tr>
<tr><td class="dcell c012">min</td></tr>
<tr><td class="dcell c012"><span class="c009">v</span> ∈ ℝ<sup><span class="c007">m</span></sup></td></tr>
</table></td><td class="dcell"> ||<span class="c009">u</span>−<span class="c009">v</span>||<sub>2</sub><sup>2</sup>    s.t.   λ<sub>1</sub>||<span class="c009">v</span>||<sub>1</sub> +λ<sub>2</sub>||<span class="c009">v</span>||<sub>2</sub><sup>2</sup>≤ τ,
    (22)</td></tr>
</table><p>
or
</p><table class="display dcenter"><tr class="c016"><td class="dcell">
</td><td class="dcell"><table class="display"><tr><td class="dcell c012">&nbsp;</td></tr>
<tr><td class="dcell c012">min</td></tr>
<tr><td class="dcell c012"><span class="c009">v</span> ∈ ℝ<sup><span class="c007">m</span></sup></td></tr>
</table></td><td class="dcell"> ||<span class="c009">u</span>−<span class="c009">v</span>||<sub>2</sub><sup>2</sup>    s.t.   λ<sub>1</sub>||<span class="c009">v</span>||<sub>1</sub> +λ<sub>2</sub>||<span class="c009">v</span>||<sub>2</sub><sup>2</sup>+ λ<sub>3</sub> <span class="c007">FL</span>(<span class="c009">v</span>) ≤ τ,
    (23)</td></tr>
</table><p>
or
</p><table class="display dcenter"><tr class="c016"><td class="dcell">
</td><td class="dcell"><table class="display"><tr><td class="dcell c012">&nbsp;</td></tr>
<tr><td class="dcell c012">min</td></tr>
<tr><td class="dcell c012"><span class="c009">v</span> ∈ ℝ<sup><span class="c007">m</span></sup></td></tr>
</table></td><td class="dcell"> </td><td class="dcell"><table class="display"><tr><td class="dcell c012">1</td></tr>
<tr><td class="hbar"></td></tr>
<tr><td class="dcell c012">2</td></tr>
</table></td><td class="dcell">||<span class="c009">u</span>−<span class="c009">v</span>||<sub>2</sub><sup>2</sup> + λ<sub>1</sub>||<span class="c009">v</span>||<sub>1</sub> +λ<sub>2</sub>||<span class="c009">v</span>||<sub>2</sub><sup>2</sup>+ λ<sub>3</sub> <span class="c007">FL</span>(<span class="c009">v</span>).
    (24)</td></tr>
</table><p>
Note that for the two last cases, the method performs a small approximation.
The method follows the regularization path, goes from one kink to another, and 
stop whenever the constraint is not satisfied anymore. The solution returned 
by the algorithm is the one obtained at the last kink of the regularization path,
which is in practice close, but not exactly the same as the solution.
This will be corrected in a future release of the toolbox.</p><table class="lstframe c011"><tr><td class="mouselstlisting"><span class="c001">% <br>
% Usage:  V=mexSparseProject(U,param);<br>
%<br>
% Name: mexSparseProject<br>
%<br>
% Description: mexSparseProject solves various optimization <br>
%     problems, including projections on a few convex sets.<br>
%     It aims at addressing the following problems<br>
%     for all columns u of U in parallel<br>
%       1) when param.mode=1 (projection on the l1-ball)<br>
%           min_v ||u-v||_2^2  s.t.  ||v||_1 &lt;= thrs<br>
%       2) when param.mode=2<br>
%           min_v ||u-v||_2^2  s.t. ||v||_2^2 + lamuda1||v||_1 &lt;= thrs<br>
%       3) when param.mode=3<br>
%           min_v ||u-v||_2^2  s.t  ||v||_1 + 0.5lamuda1||v||_2^2 &lt;= thrs <br>
%       4) when param.mode=4<br>
%           min_v 0.5||u-v||_2^2 + lamuda1||v||_1  s.t  ||v||_2^2 &lt;= thrs<br>
%       5) when param.mode=5<br>
%           min_v 0.5||u-v||_2^2 + lamuda1||v||_1 +lamuda2 FL(v) + ... <br>
%                                                   0.5lamuda_3 ||v||_2^2<br>
%          where FL denotes a "fused lasso" regularization term.<br>
%       6) when param.mode=6<br>
%          min_v ||u-v||_2^2 s.t lamuda1||v||_1 +lamuda2 FL(v) + ...<br>
%                                             0.5lamuda3||v||_2^2 &lt;= thrs<br>
%           <br>
%        When param.pos=true and param.mode &lt;= 4,<br>
%        it solves the previous problems with positivity constraints <br>
%<br>
% Inputs: U:  double m x n matrix   (input signals)<br>
%               m is the signal size<br>
%               n is the number of signals to project<br>
%         param: struct<br>
%           param.thrs (parameter)<br>
%           param.lambda1 (parameter)<br>
%           param.lambda2 (parameter)<br>
%           param.lambda3 (parameter)<br>
%           param.mode (see above)<br>
%           param.pos (optional, false by default)<br>
%           param.numThreads (optional, number of threads for exploiting<br>
%             multi-core / multi-cpus. By default, it takes the value -1,<br>
%             which automatically selects all the available CPUs/cores).<br>
%<br>
% Output: V: double m x n matrix (output matrix)<br>
%<br>
% Note: this function admits a few experimental usages, which have not<br>
%     been extensively tested:<br>
%         - single precision setting <br>
%<br>
% Author: Julien Mairal, 2009</span></td></tr>
</table><p>
The following piece of code illustrates how to use this function.
</p><table class="lstframe c011"><tr><td class="mouselstlisting"><span class="c002">clear</span> <span class="c002">all</span>;<br>
<span class="c002">randn</span>(<span class="c003">'seed'</span>,0);<br>
<span class="c001">% Data are generated</span><br>
X=<span class="c002">randn</span>(20000,100);<br>
X=X./repmat(<span class="c002">sqrt</span>(<span class="c002">sum</span>(X.^2)),[<span class="c002">size</span>(X,1) 1]);<br>
<br>
<span class="c001">% parameter of the optimization procedure are chosen</span><br>
param.numThreads=-1; <span class="c001">% number of processors/cores to use; the default choice is -1</span><br>
                    <span class="c001">% and uses all the cores of the machine</span><br>
<br>
param.pos=0;<br>
param.mode=1;       <span class="c001">% projection on the l1 ball</span><br>
param.thrs=2;<br>
<span class="c002">tic</span><br>
X1=mexSparseProject(X,param);<br>
t=<span class="c002">toc</span>;<br>
<span class="c002">toc<br>
fprintf</span>(<span class="c003">'%f signals of size %d projected per second\n'</span>,<span class="c002">size</span>(X,2)/t,<span class="c002">size</span>(X,1));<br>
<span class="c002">fprintf</span>(<span class="c003">'Checking constraint: %f, %f\n'</span>,<span class="c002">min</span>(<span class="c002">sum</span>(<span class="c002">abs</span>(X1))),<span class="c002">max</span>(<span class="c002">sum</span>(<span class="c002">abs</span>(X1))));<br>
<br>
<br>
param.mode=2;       <span class="c001">% projection on the Elastic-Net</span><br>
param.lambda1=0.15;<br>
<br>
<span class="c002">tic</span><br>
X1=mexSparseProject(X,param);<br>
t=<span class="c002">toc</span>;<br>
<span class="c002">toc<br>
fprintf</span>(<span class="c003">'%f signals of size %d projected per second\n'</span>,<span class="c002">size</span>(X,2)/t,<span class="c002">size</span>(X,1));<br>
constraints=<span class="c002">sum</span>((X1.^2))+param.lambda1*<span class="c002">sum</span>(<span class="c002">abs</span>(X1));<br>
<span class="c002">fprintf</span>(<span class="c003">'Checking constraint: %f, %f\n'</span>,<span class="c002">min</span>(constraints),<span class="c002">max</span>(constraints));<br>
<br>
param.mode=6;       <span class="c001">% projection on the FLSA</span><br>
param.lambda1=0.7;<br>
param.lambda2=0.7;<br>
param.lambda3=1.0;<br>
<br>
X=<span class="c002">rand</span>(2000,100);<br>
X=X./repmat(<span class="c002">sqrt</span>(<span class="c002">sum</span>(X.^2)),[<span class="c002">size</span>(X,1) 1]);<br>
<br>
<span class="c002">tic</span><br>
X1=mexSparseProject(X,param);<br>
t=<span class="c002">toc</span>;<br>
<span class="c002">toc<br>
fprintf</span>(<span class="c003">'%f signals of size %d projected per second\n'</span>,<span class="c002">size</span>(X,2)/t,<span class="c002">size</span>(X,1));<br>
constraints=0.5*param.lambda3*<span class="c002">sum</span>(X1.^2)+param.lambda1*<span class="c002">sum</span>(<span class="c002">abs</span>(X1))+param.lambda2*<span class="c002">sum</span>(<span class="c002">abs</span>(X1(2:<span class="c002">end</span>,:)-X1(1:<span class="c002">end</span>-1,:)));<br>
<span class="c002">fprintf</span>(<span class="c003">'Checking constraint: %f, %f\n'</span>,<span class="c002">mean</span>(constraints),<span class="c002">max</span>(constraints));<br>
<span class="c002">fprintf</span>(<span class="c003">'Projection is approximate (stops at a kink)\n'</span>,<span class="c002">mean</span>(constraints),<span class="c002">max</span>(constraints));<br>
<br>
param.mode=6;       <span class="c001">% projection on the FLSA</span><br>
param.lambda1=0.7;<br>
param.lambda2=0.7;<br>
param.lambda3=1.0;<br>
<br>
X=<span class="c002">rand</span>(2000,100);<br>
X=X./repmat(<span class="c002">sqrt</span>(<span class="c002">sum</span>(X.^2)),[<span class="c002">size</span>(X,1) 1]);<br>
<br>
<span class="c002">tic</span><br>
X1=mexSparseProject(X,param);<br>
t=<span class="c002">toc</span>;<br>
<span class="c002">toc<br>
fprintf</span>(<span class="c003">'%f signals of size %d projected per second\n'</span>,<span class="c002">size</span>(X,2)/t,<span class="c002">size</span>(X,1));<br>
constraints=0.5*param.lambda3*<span class="c002">sum</span>(X1.^2)+param.lambda1*<span class="c002">sum</span>(<span class="c002">abs</span>(X1))+param.lambda2*<span class="c002">sum</span>(<span class="c002">abs</span>(X1(2:<span class="c002">end</span>,:)-X1(1:<span class="c002">end</span>-1,:)));<br>
<span class="c002">fprintf</span>(<span class="c003">'Checking constraint: %f, %f\n'</span>,<span class="c002">mean</span>(constraints),<span class="c002">max</span>(constraints));<br>
<span class="c002">fprintf</span>(<span class="c003">'Projection is approximate (stops at a kink)\n'</span>,<span class="c002">mean</span>(constraints),<span class="c002">max</span>(constraints));</td></tr>
</table>
<!--TOC subsection id="sec22" Function mexDecompSimplex-->
<h3 class="subsection" id="sec22">4.11  Function mexDecompSimplex</h3><!--SEC END --><p>
The function implements an active-set algorithm [<a href="#ChenCVPR">37</a>] for solving
</p><table class="display dcenter"><tr class="c016"><td class="dcell">   </td><td class="dcell"><table class="display"><tr><td class="dcell c012">&nbsp;</td></tr>
<tr><td class="dcell c012">min</td></tr>
<tr><td class="dcell c012">α ∈ ℝ<sup><span class="c007">p</span></sup></td></tr>
</table></td><td class="dcell"> </td><td class="dcell"><table class="display"><tr><td class="dcell c012">1</td></tr>
<tr><td class="hbar"></td></tr>
<tr><td class="dcell c012">2</td></tr>
</table></td><td class="dcell">||<span class="c009">x</span>−<span class="c009">D</span>α||<sub>2</sub><sup>2</sup>   s.t.   α ≥ 0   and   </td><td class="dcell"><table class="display"><tr><td class="dcell c012"><span class="c007">p</span></td></tr>
<tr><td class="dcell c012"><span class="c006">∑</span></td></tr>
<tr><td class="dcell c012"><span class="c007">j</span>=1</td></tr>
</table></td><td class="dcell"> α[<span class="c007">j</span>] = 1.
</td></tr>
</table><table class="lstframe c011"><tr><td class="mouselstlisting"><span class="c001">% <br>
% Usage:  [A]=mexDecompSimplex(X,Z,param);<br>
%<br>
% Name: mexDecompSimplex<br>
%<br>
% documentation to appear soon<br>
%<br>
% Inputs: X:  double m x n matrix   (input signals)<br>
%               m is the signal size<br>
%               n is the number of signals to decompose<br>
% Output: Z: double %<br>
%<br>
% Author: Yuansi Chen and Julien Mairal, 2014</span></td></tr>
</table>
<!--TOC section id="sec23" Proximal Toolbox-->
<h2 class="section" id="sec23">5  Proximal Toolbox</h2><!--SEC END --><p>
The previous toolbox we have presented is well
adapted for solving a large number of small and medium-scale sparse
decomposition problems with the square loss, which is typical from the
classical dictionary learning framework. We now present
a new software package that is adapted for solving a wide range of
possibly large-scale learning problems, with several combinations of losses and
regularization terms. The method implements the proximal methods
of [<a href="#beck">1</a>], and includes the proximal solvers for the tree-structured
regularization of [<a href="#jenatton3">15</a>], and the solver of [<a href="#mairal10">22</a>] for
general structured sparse regularization. 
The solver for structured sparse regularization norms includes a C++ max-flow
implementation of the push-relabel algorithm of [<a href="#goldberg">13</a>], with
heuristics proposed by [<a href="#cherkassky">5</a>].</p><p>This implementation also provides robust stopping criteria based on
<em>duality gaps</em>, which are presented in Appendix <a href="#appendix">A</a>. It can handle intercepts (unregularized variables). The general formulation that our software
can solve take the form
</p><table class="display dcenter"><tr class="c016"><td class="dcell"><table class="display"><tr><td class="dcell c012">&nbsp;</td></tr>
<tr><td class="dcell c012">min</td></tr>
<tr><td class="dcell c012"><span class="c009">w</span> ∈ ℝ<sup><span class="c007">p</span></sup></td></tr>
</table></td><td class="dcell"> [<span class="c007">g</span>(<span class="c009">w</span>) </td><td class="dcell"><table class="display"><tr><td class="dcell c012"><span class="c005">▵</span></td></tr>
<tr><td class="dcell c012">=</td></tr>
<tr><td class="dcell c012">&nbsp;</td></tr>
</table></td><td class="dcell"> <span class="c007">f</span>(<span class="c009">w</span>) + λψ(<span class="c009">w</span>)],
</td></tr>
</table><p>
where <span class="c007">f</span> is a smooth loss function and ψ is a regularization function.
When one optimizes a matrix <span class="c009">W</span> in ℝ<sup><span class="c007">p</span> × <span class="c007">r</span></sup> instead of
a vector <span class="c009">w</span> in ℝ<sup><span class="c007">p</span></sup>, we will write 
</p><table class="display dcenter"><tr class="c016"><td class="dcell"><table class="display"><tr><td class="dcell c012">&nbsp;</td></tr>
<tr><td class="dcell c012">min</td></tr>
<tr><td class="dcell c012"><span class="c009">W</span> ∈ ℝ<sup><span class="c007">p</span> × <span class="c007">r</span></sup></td></tr>
</table></td><td class="dcell"> [<span class="c007">g</span>(<span class="c009">W</span>) </td><td class="dcell"><table class="display"><tr><td class="dcell c012"><span class="c005">▵</span></td></tr>
<tr><td class="dcell c012">=</td></tr>
<tr><td class="dcell c012">&nbsp;</td></tr>
</table></td><td class="dcell"> <span class="c007">f</span>(<span class="c009">W</span>) + λψ(<span class="c009">W</span>)].
</td></tr>
</table><p>
Note that the software can possibly handle nonnegativity constraints.</p><p>We start by presenting the type of regularization implemented in the software
</p>
<!--TOC subsection id="sec24" Regularization Functions-->
<h3 class="subsection" id="sec24">5.1  Regularization Functions</h3><!--SEC END --><p>
Our software can handle the following regularization functions ψ for vectors <span class="c009">w</span> in ℝ<sup><span class="c007">p</span></sup>:
</p><ul class="itemize"><li class="li-itemize">
<span class="c010">The Tikhonov regularization</span>: ψ(<span class="c009">w</span>) =<sup><span class="c005">▵</span></sup> 1/2||<span class="c009">w</span>||<sub>2</sub><sup>2</sup>.
</li><li class="li-itemize"><span class="c010">The </span>ℓ<sub>1</sub><span class="c010">-norm</span>: ψ(<span class="c009">w</span>) =<sup><span class="c005">▵</span></sup> ||<span class="c009">w</span>||<sub>1</sub>.
</li><li class="li-itemize"><span class="c010">The Elastic-Net</span>: ψ(<span class="c009">w</span>) =<sup><span class="c005">▵</span></sup> ||<span class="c009">w</span>||<sub>1</sub>+γ||<span class="c009">w</span>||<sub>2</sub><sup>2</sup>.
</li><li class="li-itemize"><span class="c010">The Fused-Lasso</span>: ψ(<span class="c009">w</span>) =<sup><span class="c005">▵</span></sup> ||<span class="c009">w</span>||<sub>1</sub>+γ||<span class="c009">w</span>||<sub>2</sub><sup>2</sup>+γ<sub>2</sub>∑<sub><span class="c007">i</span>=1</sub><sup><span class="c007">p</span>−1</sup>|<span class="c009">w</span><sub><span class="c007">i</span>+1</sub>−<span class="c009">w</span><sub><span class="c007">i</span></sub>|.
</li><li class="li-itemize"><span class="c010">The group Lasso</span>: ψ(<span class="c009">w</span>) =<sup><span class="c005">▵</span></sup> ∑<sub><span class="c007">g</span> ∈ <span class="c008">G</span></sub> η<sub><span class="c007">g</span></sub> ||<span class="c009">w</span><sub><span class="c007">g</span></sub>||<sub>2</sub>, where <span class="c008">G</span> are groups of variables.
</li><li class="li-itemize"><span class="c010">The group Lasso with </span>ℓ<sub>∞</sub><span class="c010">-norm</span>: ψ(<span class="c009">w</span>) =<sup><span class="c005">▵</span></sup> ∑<sub><span class="c007">g</span> ∈ <span class="c008">G</span></sub> η<sub><span class="c007">g</span></sub> ||<span class="c009">w</span><sub><span class="c007">g</span></sub>||<sub>∞</sub>, where <span class="c008">G</span> are groups of variables.
</li><li class="li-itemize"><span class="c010">The sparse group Lasso</span>: same as above but with an additional ℓ<sub>1</sub> term.
</li><li class="li-itemize"><span class="c010">The tree-structured sum of </span>ℓ<sub>2</sub><span class="c010">-norms</span>: ψ(<span class="c009">w</span>) =<sup><span class="c005">▵</span></sup> ∑<sub><span class="c007">g</span> ∈ <span class="c008">G</span></sub> η<sub><span class="c007">g</span></sub> ||<span class="c009">w</span><sub><span class="c007">g</span></sub>||<sub>2</sub>, where <span class="c008">G</span> is a tree-structured set of groups [<a href="#jenatton3">15</a>], and the η<sub><span class="c007">g</span></sub> are positive weights.
</li><li class="li-itemize"><span class="c010">The tree-structured sum of </span>ℓ<sub>∞</sub><span class="c010">-norms</span>: ψ(<span class="c009">w</span>) =<sup><span class="c005">▵</span></sup> ∑<sub><span class="c007">g</span> ∈ <span class="c008">G</span></sub> η<sub><span class="c007">g</span></sub> ||<span class="c009">w</span><sub><span class="c007">g</span></sub>||<sub>∞</sub>. See [<a href="#jenatton3">15</a>]
</li><li class="li-itemize"><span class="c010">General sum of </span>ℓ<sub>∞</sub><span class="c010">-norms</span>: ψ(<span class="c009">w</span>) =<sup><span class="c005">▵</span></sup> ∑<sub><span class="c007">g</span> ∈ <span class="c008">G</span></sub> η<sub><span class="c007">g</span></sub> ||<span class="c009">w</span><sub><span class="c007">g</span></sub>||<sub>∞</sub>, where no assumption are made on the groups <span class="c008">G</span>.
</li><li class="li-itemize"><span class="c010">The path-coding penalties</span> of [<a href="#mairal14">24</a>].
</li><li class="li-itemize"><span class="c010">the </span>ℓ<sub>1</sub><span class="c010">-constraint</span>.
</li><li class="li-itemize"><span class="c010">a few undocumented ones</span>.
</li></ul><p>
Our software also handles regularization functions ψ on matrices <span class="c009">W</span> in ℝ<sup><span class="c007">p</span> × <span class="c007">r</span></sup> (note that <span class="c009">W</span> can be transposed in these formulations). In particular,
</p><ul class="itemize"><li class="li-itemize">
<span class="c010">The </span>ℓ<sub>1</sub>/ℓ<sub>2</sub><span class="c010">-norm</span>: ψ(<span class="c009">W</span>) =<sup><span class="c005">▵</span></sup> ∑<sub><span class="c007">i</span>=1</sub><sup><span class="c007">p</span></sup> ||<span class="c009">W</span><sub><span class="c007">i</span></sub>||<sub>2</sub>, where <span class="c009">W</span><sub><span class="c007">i</span></sub> denotes the <span class="c007">i</span>-th row of <span class="c009">W</span>.
</li><li class="li-itemize"><span class="c010">The </span>ℓ<sub>1</sub>/ℓ<sub>∞</sub><span class="c010">-norm</span>: ψ(<span class="c009">W</span>) =<sup><span class="c005">▵</span></sup> ∑<sub><span class="c007">i</span>=1</sub><sup><span class="c007">p</span></sup> ||<span class="c009">W</span><sub><span class="c007">i</span></sub>||<sub>∞</sub>,
</li><li class="li-itemize"><span class="c010">The </span>ℓ<sub>1</sub>/ℓ<sub>2</sub><span class="c010">+</span>ℓ<sub>1</sub><span class="c010">-norm</span>: ψ(<span class="c009">W</span>) =<sup><span class="c005">▵</span></sup> ∑<sub><span class="c007">i</span>=1</sub><sup><span class="c007">p</span></sup> ||<span class="c009">W</span><sub><span class="c007">i</span></sub>||<sub>2</sub> + λ<sub>2</sub> ∑<sub><span class="c007">i</span>,<span class="c007">j</span></sub>|<span class="c009">W</span><sub><span class="c007">ij</span></sub>|.
</li><li class="li-itemize"><span class="c010">The </span>ℓ<sub>1</sub>/ℓ<sub>∞</sub><span class="c010">+</span>ℓ<sub>1</sub><span class="c010">-norm</span>: ψ(<span class="c009">W</span>) =<sup><span class="c005">▵</span></sup> ∑<sub><span class="c007">i</span>=1</sub><sup><span class="c007">p</span></sup> ||<span class="c009">W</span><sub><span class="c007">i</span></sub>||<sub>∞</sub>+λ<sub>2</sub> ∑<sub><span class="c007">i</span>,<span class="c007">j</span></sub>|<span class="c009">W</span><sub><span class="c007">ij</span></sub>|,
</li><li class="li-itemize"><span class="c010">The </span>ℓ<sub>1</sub>/ℓ<sub>∞</sub><span class="c010">-norm on rows and columns</span>: ψ(<span class="c009">W</span>) =<sup><span class="c005">▵</span></sup> ∑<sub><span class="c007">i</span>=1</sub><sup><span class="c007">p</span></sup> ||<span class="c009">W</span><sub><span class="c007">i</span></sub>||<sub>∞</sub>+λ<sub>2</sub> ∑<sub><span class="c007">j</span>=1</sub><sup><span class="c007">r</span></sup>||<span class="c009">W</span><sup><span class="c007">j</span></sup>||<sub>∞</sub>, where <span class="c009">W</span><sup><span class="c007">j</span></sup> denotes the <span class="c007">j</span>-th column of <span class="c009">W</span>.
</li><li class="li-itemize"><span class="c010">The multi-task tree-structured sum of </span>ℓ<sub>∞</sub><span class="c010">-norms</span>: 
<table class="display dcenter"><tr class="c016"><td class="dcell">
ψ(<span class="c009">W</span>)</td><td class="dcell"><table class="display"><tr><td class="dcell c012"><span class="c005">▵</span></td></tr>
<tr><td class="dcell c012">=</td></tr>
<tr><td class="dcell c012">&nbsp;</td></tr>
</table></td><td class="dcell"> </td><td class="dcell"><table class="display"><tr><td class="dcell c012"><span class="c007">r</span></td></tr>
<tr><td class="dcell c012"><span class="c006">∑</span></td></tr>
<tr><td class="dcell c012"><span class="c007">i</span>=1</td></tr>
</table></td><td class="dcell"> </td><td class="dcell"><table class="display"><tr><td class="dcell c012">&nbsp;</td></tr>
<tr><td class="dcell c012"><span class="c006">∑</span></td></tr>
<tr><td class="dcell c012"><span class="c007">g</span> ∈ <span class="c008">G</span></td></tr>
</table></td><td class="dcell"> η<sub><span class="c007">g</span></sub>||<span class="c009">w</span><sub><span class="c007">g</span></sub><sup><span class="c007">i</span></sup>||<sub>∞</sub>+ γ </td><td class="dcell"><table class="display"><tr><td class="dcell c012">&nbsp;</td></tr>
<tr><td class="dcell c012"><span class="c006">∑</span></td></tr>
<tr><td class="dcell c012"><span class="c007">g</span> ∈ <span class="c008">G</span></td></tr>
</table></td><td class="dcell"> η<sub><span class="c007">g</span></sub> </td><td class="dcell"><table class="display"><tr><td class="dcell c012">&nbsp;</td></tr>
<tr><td class="dcell c012">max</td></tr>
<tr><td class="dcell c012"><span class="c007">j</span> ∈ <span class="c007">g</span></td></tr>
</table></td><td class="dcell">||<span class="c009">W</span><sub><span class="c007">j</span></sub>||<sub>∞</sub>,
<a id="software:eq:struct"></a>
    (25)</td></tr>
</table>
where the first double sums is in fact a sum of independent structured norms on the columns <span class="c009">w</span><sup><span class="c007">i</span></sup> of <span class="c009">W</span>, and the right term is a tree-structured regularization norm applied to the ℓ<sub>∞</sub>-norm of the rows of <span class="c009">W</span>, thereby inducing the tree-structured regularization at the row level. <span class="c008">G</span> is here a tree-structured set of groups.
</li><li class="li-itemize"><span class="c010">The multi-task general sum of </span>ℓ<sub>∞</sub><span class="c010">-norms</span> is the same as Eq. (<a href="#software%3Aeq%3Astruct">25</a>) except that the groups <span class="c008">G</span> are general overlapping groups.
</li><li class="li-itemize"><span class="c010">The trace norm</span>: ψ(<span class="c009">W</span>) =<sup><span class="c005">▵</span></sup> ||<span class="c009">W</span>||<sub>*</sub>.
</li></ul><p>
Non-convex regularizations are also implemented with the ISTA algorithm (no duality gaps are of course provided in these cases):
</p><ul class="itemize"><li class="li-itemize">
<span class="c010">The </span>ℓ<sub>0</sub><span class="c010">-pseudo-norm</span>: ψ(<span class="c009">w</span>) =<sup><span class="c005">▵</span></sup> ||<span class="c009">w</span>||<sub>0</sub>.
</li><li class="li-itemize"><span class="c010">The rank</span>: ψ(<span class="c009">W</span>) =<sup><span class="c005">▵</span></sup> randk(<span class="c009">W</span>).
</li><li class="li-itemize"><span class="c010">The tree-structured </span>ℓ<sub>0</sub><span class="c010">-pseudo-norm</span>: ψ(<span class="c009">w</span>) =<sup><span class="c005">▵</span></sup> ∑<sub><span class="c007">g</span> ∈ <span class="c008">G</span></sub> δ<sub><span class="c009">w</span><sub><span class="c007">g</span></sub> ≠ 0</sub>.
</li></ul><p>All of these regularization terms for vectors or matrices can be coupled with
nonnegativity constraints. It is also possible to add an intercept, which one
wishes not to regularize, and we will include this possibility in the next
sections. There are also a few hidden undocumented options which are available in the source code.</p><p>We now present 3 functions for computing proximal operators associated to the previous regularization functions.
</p>
<!--TOC subsection id="sec25" Function mexProximalFlat-->
<h3 class="subsection" id="sec25">5.2  Function mexProximalFlat</h3><!--SEC END --><p>
This function computes the proximal operators associated to many regularization functions, for input signals <span class="c009">U</span>=[<span class="c009">u</span><sup>1</sup>,…,<span class="c009">u</span><sup><span class="c007">n</span></sup>] in ℝ<sup><span class="c007">p</span> × <span class="c007">n</span></sup>, it finds a matrix <span class="c009">V</span>=[<span class="c009">v</span><sup>1</sup>,…,<span class="c009">v</span><sup><span class="c007">n</span></sup>] in ℝ<sup><span class="c007">p</span> × <span class="c007">n</span></sup> such that:</p><p>•  If one chooses a regularization function on vectors, for every column <span class="c009">u</span> of <span class="c009">U</span>, it computes one column <span class="c009">v</span> of <span class="c009">V</span> solving
</p><table class="display dcenter"><tr class="c016"><td class="dcell">
</td><td class="dcell"><table class="display"><tr><td class="dcell c012">&nbsp;</td></tr>
<tr><td class="dcell c012">min</td></tr>
<tr><td class="dcell c012"><span class="c009">v</span> ∈ ℝ<sup><span class="c007">p</span></sup></td></tr>
</table></td><td class="dcell"> </td><td class="dcell"><table class="display"><tr><td class="dcell c012">1</td></tr>
<tr><td class="hbar"></td></tr>
<tr><td class="dcell c012">2</td></tr>
</table></td><td class="dcell">||<span class="c009">u</span>−<span class="c009">v</span>||<sub>2</sub><sup>2</sup> + λ ||<span class="c009">v</span>||<sub>0</sub>,
    (26)</td></tr>
</table><p>
or
</p><table class="display dcenter"><tr class="c016"><td class="dcell">
</td><td class="dcell"><table class="display"><tr><td class="dcell c012">&nbsp;</td></tr>
<tr><td class="dcell c012">min</td></tr>
<tr><td class="dcell c012"><span class="c009">v</span> ∈ ℝ<sup><span class="c007">p</span></sup></td></tr>
</table></td><td class="dcell"> </td><td class="dcell"><table class="display"><tr><td class="dcell c012">1</td></tr>
<tr><td class="hbar"></td></tr>
<tr><td class="dcell c012">2</td></tr>
</table></td><td class="dcell">||<span class="c009">u</span>−<span class="c009">v</span>||<sub>2</sub><sup>2</sup> + λ ||<span class="c009">v</span>||<sub>1</sub>,
    (27)</td></tr>
</table><p>
or
</p><table class="display dcenter"><tr class="c016"><td class="dcell">
</td><td class="dcell"><table class="display"><tr><td class="dcell c012">&nbsp;</td></tr>
<tr><td class="dcell c012">min</td></tr>
<tr><td class="dcell c012"><span class="c009">v</span> ∈ ℝ<sup><span class="c007">p</span></sup></td></tr>
</table></td><td class="dcell"> </td><td class="dcell"><table class="display"><tr><td class="dcell c012">1</td></tr>
<tr><td class="hbar"></td></tr>
<tr><td class="dcell c012">2</td></tr>
</table></td><td class="dcell">||<span class="c009">u</span>−<span class="c009">v</span>||<sub>2</sub><sup>2</sup> + </td><td class="dcell"><table class="display"><tr><td class="dcell c012">λ</td></tr>
<tr><td class="hbar"></td></tr>
<tr><td class="dcell c012">2</td></tr>
</table></td><td class="dcell"> ||<span class="c009">v</span>||<sub>2</sub><sup>2</sup>,
    (28)</td></tr>
</table><p>
or
</p><table class="display dcenter"><tr class="c016"><td class="dcell">
</td><td class="dcell"><table class="display"><tr><td class="dcell c012">&nbsp;</td></tr>
<tr><td class="dcell c012">min</td></tr>
<tr><td class="dcell c012"><span class="c009">v</span> ∈ ℝ<sup><span class="c007">p</span></sup></td></tr>
</table></td><td class="dcell"> </td><td class="dcell"><table class="display"><tr><td class="dcell c012">1</td></tr>
<tr><td class="hbar"></td></tr>
<tr><td class="dcell c012">2</td></tr>
</table></td><td class="dcell">||<span class="c009">u</span>−<span class="c009">v</span>||<sub>2</sub><sup>2</sup> + λ ||<span class="c009">v</span>||<sub>1</sub> + λ<sub>2</sub>||<span class="c009">v</span>||<sub>2</sub><sup>2</sup>,
    (29)</td></tr>
</table><p>
or
</p><table class="display dcenter"><tr class="c016"><td class="dcell">
</td><td class="dcell"><table class="display"><tr><td class="dcell c012">&nbsp;</td></tr>
<tr><td class="dcell c012">min</td></tr>
<tr><td class="dcell c012"><span class="c009">v</span> ∈ ℝ<sup><span class="c007">p</span></sup></td></tr>
</table></td><td class="dcell"> </td><td class="dcell"><table class="display"><tr><td class="dcell c012">1</td></tr>
<tr><td class="hbar"></td></tr>
<tr><td class="dcell c012">2</td></tr>
</table></td><td class="dcell">||<span class="c009">u</span>−<span class="c009">v</span>||<sub>2</sub><sup>2</sup> + λ</td><td class="dcell"><table class="display"><tr><td class="dcell c012"><span class="c007">p</span>−1</td></tr>
<tr><td class="dcell c012"><span class="c006">∑</span></td></tr>
<tr><td class="dcell c012"><span class="c007">j</span>=1</td></tr>
</table></td><td class="dcell">|<span class="c009">v</span><sub><span class="c007">j</span>+1</sub><sup><span class="c007">i</span></sup>−<span class="c009">v</span><sub><span class="c007">j</span></sub><sup><span class="c007">i</span></sup>|+λ<sub>2</sub> ||<span class="c009">v</span>||<sub>1</sub> + λ<sub>3</sub>||<span class="c009">v</span>||<sub>2</sub><sup>2</sup>,
    (30)</td></tr>
</table><p>
or
</p><table class="display dcenter"><tr class="c016"><td class="dcell">
</td><td class="dcell"><table class="display"><tr><td class="dcell c012">&nbsp;</td></tr>
<tr><td class="dcell c012">min</td></tr>
<tr><td class="dcell c012"><span class="c009">v</span> ∈ ℝ<sup><span class="c007">p</span></sup></td></tr>
</table></td><td class="dcell"> </td><td class="dcell"><table class="display"><tr><td class="dcell c012">1</td></tr>
<tr><td class="hbar"></td></tr>
<tr><td class="dcell c012">2</td></tr>
</table></td><td class="dcell">||<span class="c009">u</span>−<span class="c009">v</span>||<sub>2</sub><sup>2</sup> + λ </td><td class="dcell"><table class="display"><tr><td class="dcell c012">&nbsp;</td></tr>
<tr><td class="dcell c012"><span class="c006">∑</span></td></tr>
<tr><td class="dcell c012"><span class="c007">g</span> ∈ <span class="c008">T</span></td></tr>
</table></td><td class="dcell"> δ<sup><span class="c007">g</span></sup>(<span class="c009">v</span>),
    (31)</td></tr>
</table><p>
where <span class="c008">T</span> is a tree-structured set of groups (see [<a href="#jenatton4">16</a>]), and δ<sup><span class="c007">g</span></sup>(<span class="c009">v</span>) = 0 if <span class="c009">v</span><sub><span class="c007">g</span></sub>=0 and 1 otherwise.
It can also solve
</p><table class="display dcenter"><tr class="c016"><td class="dcell">
</td><td class="dcell"><table class="display"><tr><td class="dcell c012">&nbsp;</td></tr>
<tr><td class="dcell c012">min</td></tr>
<tr><td class="dcell c012"><span class="c009">v</span> ∈ ℝ<sup><span class="c007">p</span></sup></td></tr>
</table></td><td class="dcell"> </td><td class="dcell"><table class="display"><tr><td class="dcell c012">1</td></tr>
<tr><td class="hbar"></td></tr>
<tr><td class="dcell c012">2</td></tr>
</table></td><td class="dcell">||<span class="c009">u</span>−<span class="c009">v</span>||<sub>2</sub><sup>2</sup> + λ </td><td class="dcell"><table class="display"><tr><td class="dcell c012">&nbsp;</td></tr>
<tr><td class="dcell c012"><span class="c006">∑</span></td></tr>
<tr><td class="dcell c012"><span class="c007">g</span> ∈ <span class="c008">T</span></td></tr>
</table></td><td class="dcell"> η<sup><span class="c007">g</span></sup> ||<span class="c009">v</span><sub><span class="c007">g</span></sub>||<sub>2</sub>,
    (32)</td></tr>
</table><p>
or
</p><table class="display dcenter"><tr class="c016"><td class="dcell">
</td><td class="dcell"><table class="display"><tr><td class="dcell c012">&nbsp;</td></tr>
<tr><td class="dcell c012">min</td></tr>
<tr><td class="dcell c012"><span class="c009">v</span> ∈ ℝ<sup><span class="c007">p</span></sup></td></tr>
</table></td><td class="dcell"> </td><td class="dcell"><table class="display"><tr><td class="dcell c012">1</td></tr>
<tr><td class="hbar"></td></tr>
<tr><td class="dcell c012">2</td></tr>
</table></td><td class="dcell">||<span class="c009">u</span>−<span class="c009">v</span>||<sub>2</sub><sup>2</sup> + λ </td><td class="dcell"><table class="display"><tr><td class="dcell c012">&nbsp;</td></tr>
<tr><td class="dcell c012"><span class="c006">∑</span></td></tr>
<tr><td class="dcell c012"><span class="c007">g</span> ∈ <span class="c008">T</span></td></tr>
</table></td><td class="dcell"> η<sup><span class="c007">g</span></sup> ||<span class="c009">v</span><sub><span class="c007">g</span></sub>||<sub>∞</sub>,
    (33)</td></tr>
</table><p>
or
</p><table class="display dcenter"><tr class="c016"><td class="dcell">
</td><td class="dcell"><table class="display"><tr><td class="dcell c012">&nbsp;</td></tr>
<tr><td class="dcell c012">min</td></tr>
<tr><td class="dcell c012"><span class="c009">v</span> ∈ ℝ<sup><span class="c007">p</span></sup></td></tr>
</table></td><td class="dcell"> </td><td class="dcell"><table class="display"><tr><td class="dcell c012">1</td></tr>
<tr><td class="hbar"></td></tr>
<tr><td class="dcell c012">2</td></tr>
</table></td><td class="dcell">||<span class="c009">u</span>−<span class="c009">v</span>||<sub>2</sub><sup>2</sup> + λ </td><td class="dcell"><table class="display"><tr><td class="dcell c012">&nbsp;</td></tr>
<tr><td class="dcell c012"><span class="c006">∑</span></td></tr>
<tr><td class="dcell c012"><span class="c007">g</span> ∈ <span class="c008">G</span></td></tr>
</table></td><td class="dcell"> η<sup><span class="c007">g</span></sup> ||<span class="c009">v</span><sub><span class="c007">g</span></sub>||<sub>∞</sub>,
    (34)</td></tr>
</table><p>
where <span class="c008">G</span> is any kind of set of groups.</p><p>This function can also solve the following proximal operators on matrices
</p><table class="display dcenter"><tr class="c016"><td class="dcell">
</td><td class="dcell"><table class="display"><tr><td class="dcell c012">&nbsp;</td></tr>
<tr><td class="dcell c012">min</td></tr>
<tr><td class="dcell c012"><span class="c009">V</span> ∈ ℝ<sup><span class="c007">p</span> × <span class="c007">n</span></sup></td></tr>
</table></td><td class="dcell"> </td><td class="dcell"><table class="display"><tr><td class="dcell c012">1</td></tr>
<tr><td class="hbar"></td></tr>
<tr><td class="dcell c012">2</td></tr>
</table></td><td class="dcell">||<span class="c009">U</span>−<span class="c009">V</span>||<sub><span class="c007">F</span></sub><sup>2</sup> + λ </td><td class="dcell"><table class="display"><tr><td class="dcell c012"><span class="c007">p</span></td></tr>
<tr><td class="dcell c012"><span class="c006">∑</span></td></tr>
<tr><td class="dcell c012"><span class="c007">i</span>=1</td></tr>
</table></td><td class="dcell"> ||<span class="c009">V</span><sub><span class="c007">i</span></sub>||<sub>2</sub>, 
    (35)</td></tr>
</table><p>
where <span class="c009">V</span><sub><span class="c007">i</span></sub> is the <span class="c007">i</span>-th row of <span class="c009">V</span>, or
</p><table class="display dcenter"><tr class="c016"><td class="dcell">
</td><td class="dcell"><table class="display"><tr><td class="dcell c012">&nbsp;</td></tr>
<tr><td class="dcell c012">min</td></tr>
<tr><td class="dcell c012"><span class="c009">V</span> ∈ ℝ<sup><span class="c007">p</span> × <span class="c007">n</span></sup></td></tr>
</table></td><td class="dcell"> </td><td class="dcell"><table class="display"><tr><td class="dcell c012">1</td></tr>
<tr><td class="hbar"></td></tr>
<tr><td class="dcell c012">2</td></tr>
</table></td><td class="dcell">||<span class="c009">U</span>−<span class="c009">V</span>||<sub><span class="c007">F</span></sub><sup>2</sup> + λ </td><td class="dcell"><table class="display"><tr><td class="dcell c012"><span class="c007">p</span></td></tr>
<tr><td class="dcell c012"><span class="c006">∑</span></td></tr>
<tr><td class="dcell c012"><span class="c007">i</span>=1</td></tr>
</table></td><td class="dcell"> ||<span class="c009">V</span><sub><span class="c007">i</span></sub>||<sub>∞</sub>, 
    (36)</td></tr>
</table><p>
or
</p><table class="display dcenter"><tr class="c016"><td class="dcell">
</td><td class="dcell"><table class="display"><tr><td class="dcell c012">&nbsp;</td></tr>
<tr><td class="dcell c012">min</td></tr>
<tr><td class="dcell c012"><span class="c009">V</span> ∈ ℝ<sup><span class="c007">p</span> × <span class="c007">n</span></sup></td></tr>
</table></td><td class="dcell"> </td><td class="dcell"><table class="display"><tr><td class="dcell c012">1</td></tr>
<tr><td class="hbar"></td></tr>
<tr><td class="dcell c012">2</td></tr>
</table></td><td class="dcell">||<span class="c009">U</span>−<span class="c009">V</span>||<sub><span class="c007">F</span></sub><sup>2</sup> + λ </td><td class="dcell"><table class="display"><tr><td class="dcell c012"><span class="c007">p</span></td></tr>
<tr><td class="dcell c012"><span class="c006">∑</span></td></tr>
<tr><td class="dcell c012"><span class="c007">i</span>=1</td></tr>
</table></td><td class="dcell"> ||<span class="c009">V</span><sub><span class="c007">i</span></sub>||<sub>2</sub> +λ<sub>2</sub> </td><td class="dcell"><table class="display"><tr><td class="dcell c012"><span class="c007">p</span></td></tr>
<tr><td class="dcell c012"><span class="c006">∑</span></td></tr>
<tr><td class="dcell c012"><span class="c007">i</span>=1</td></tr>
</table></td><td class="dcell"><table class="display"><tr><td class="dcell c012"><span class="c007">n</span></td></tr>
<tr><td class="dcell c012"><span class="c006">∑</span></td></tr>
<tr><td class="dcell c012"><span class="c007">j</span>=1</td></tr>
</table></td><td class="dcell"> |<span class="c009">V</span><sub><span class="c007">ij</span></sub>|, 
    (37)</td></tr>
</table><p>
or
</p><table class="display dcenter"><tr class="c016"><td class="dcell">
</td><td class="dcell"><table class="display"><tr><td class="dcell c012">&nbsp;</td></tr>
<tr><td class="dcell c012">min</td></tr>
<tr><td class="dcell c012"><span class="c009">V</span> ∈ ℝ<sup><span class="c007">p</span> × <span class="c007">n</span></sup></td></tr>
</table></td><td class="dcell"> </td><td class="dcell"><table class="display"><tr><td class="dcell c012">1</td></tr>
<tr><td class="hbar"></td></tr>
<tr><td class="dcell c012">2</td></tr>
</table></td><td class="dcell">||<span class="c009">U</span>−<span class="c009">V</span>||<sub><span class="c007">F</span></sub><sup>2</sup> + λ </td><td class="dcell"><table class="display"><tr><td class="dcell c012"><span class="c007">p</span></td></tr>
<tr><td class="dcell c012"><span class="c006">∑</span></td></tr>
<tr><td class="dcell c012"><span class="c007">i</span>=1</td></tr>
</table></td><td class="dcell"> ||<span class="c009">V</span><sub><span class="c007">i</span></sub>||<sub>∞</sub>+λ<sub>2</sub> </td><td class="dcell"><table class="display"><tr><td class="dcell c012"><span class="c007">p</span></td></tr>
<tr><td class="dcell c012"><span class="c006">∑</span></td></tr>
<tr><td class="dcell c012"><span class="c007">i</span>=1</td></tr>
</table></td><td class="dcell"><table class="display"><tr><td class="dcell c012"><span class="c007">n</span></td></tr>
<tr><td class="dcell c012"><span class="c006">∑</span></td></tr>
<tr><td class="dcell c012"><span class="c007">j</span>=1</td></tr>
</table></td><td class="dcell"> |<span class="c009">V</span><sub><span class="c007">ij</span></sub>|, 
    (38)</td></tr>
</table><p>
or
</p><table class="display dcenter"><tr class="c016"><td class="dcell">
</td><td class="dcell"><table class="display"><tr><td class="dcell c012">&nbsp;</td></tr>
<tr><td class="dcell c012">min</td></tr>
<tr><td class="dcell c012"><span class="c009">V</span> ∈ ℝ<sup><span class="c007">p</span> × <span class="c007">n</span></sup></td></tr>
</table></td><td class="dcell"> </td><td class="dcell"><table class="display"><tr><td class="dcell c012">1</td></tr>
<tr><td class="hbar"></td></tr>
<tr><td class="dcell c012">2</td></tr>
</table></td><td class="dcell">||<span class="c009">U</span>−<span class="c009">V</span>||<sub><span class="c007">F</span></sub><sup>2</sup> + λ </td><td class="dcell"><table class="display"><tr><td class="dcell c012"><span class="c007">p</span></td></tr>
<tr><td class="dcell c012"><span class="c006">∑</span></td></tr>
<tr><td class="dcell c012"><span class="c007">i</span>=1</td></tr>
</table></td><td class="dcell"> ||<span class="c009">V</span><sub><span class="c007">i</span></sub>||<sub>∞</sub>+λ<sub>2</sub> </td><td class="dcell"><table class="display"><tr><td class="dcell c012"><span class="c007">n</span></td></tr>
<tr><td class="dcell c012"><span class="c006">∑</span></td></tr>
<tr><td class="dcell c012"><span class="c007">j</span>=1</td></tr>
</table></td><td class="dcell"> ||<span class="c009">V</span><sup><span class="c007">j</span></sup>||<sub>∞</sub>.
    (39)</td></tr>
</table><p>
where <span class="c009">V</span><sup><span class="c007">j</span></sup> is the <span class="c007">j</span>-th column of <span class="c009">V</span>.</p><p>See details below:
</p><table class="lstframe c011"><tr><td class="mouselstlisting"><span class="c001">% <br>
% Usage:  [V [val_regularizer]]=mexProximalFlat(U,param);<br>
%<br>
% Name: mexProximalFlat<br>
%<br>
% Description: mexProximalFlat computes proximal operators. Depending<br>
%         on the value of param.regul, it computes <br>
%<br>
%         Given an input matrix U=[u^1,\ldots,u^n], it computes a matrix <br>
%         V=[v^1,\ldots,v^n] such that<br>
%         if one chooses a regularization functions on vectors, it computes<br>
%         for each column u of U, a column v of V solving<br>
%         if param.regul='l0'<br>
%             argmin 0.5||u-v||_2^2 + lambda||v||_0<br>
%         if param.regul='l1'<br>
%             argmin 0.5||u-v||_2^2 + lambda||v||_1<br>
%         if param.regul='l2'<br>
%             argmin 0.5||u-v||_2^2 + 0.5lambda||v||_2^2<br>
%         if param.regul='elastic-net'<br>
%             argmin 0.5||u-v||_2^2 + lambda||v||_1 + lambda_2||v||_2^2<br>
%         if param.regul='fused-lasso'<br>
%             argmin 0.5||u-v||_2^2 + lambda FL(v) + ...<br>
%                               ...  lambda_2||v||_1 + lambda_3||v||_2^2<br>
%         if param.regul='linf'<br>
%             argmin 0.5||u-v||_2^2 + lambda||v||_inf<br>
%         if param.regul='l1-constraint'<br>
%             argmin 0.5||u-v||_2^2 s.t. ||v||_1 &lt;= lambda<br>
%         if param.regul='l2-not-squared'<br>
%             argmin 0.5||u-v||_2^2 + lambda||v||_2<br>
%         if param.regul='group-lasso-l2'  <br>
%             argmin 0.5||u-v||_2^2 + lambda sum_g ||v_g||_2 <br>
%             where the groups are either defined by param.groups or by param.size_group,<br>
%         if param.regul='group-lasso-linf'<br>
%             argmin 0.5||u-v||_2^2 + lambda sum_g ||v_g||_inf<br>
%         if param.regul='sparse-group-lasso-l2'  <br>
%             argmin 0.5||u-v||_2^2 + lambda sum_g ||v_g||_2 + lambda_2 ||v||_1<br>
%             where the groups are either defined by param.groups or by param.size_group,<br>
%         if param.regul='sparse-group-lasso-linf'<br>
%             argmin 0.5||u-v||_2^2 + lambda sum_g ||v_g||_inf + lambda_2 ||v||_1<br>
%         if param.regul='trace-norm-vec' <br>
%             argmin 0.5||u-v||_2^2 + lambda ||mat(v)||_* <br>
%            where mat(v) has param.size_group rows<br>
%<br>
%         if one chooses a regularization function on matrices<br>
%         if param.regul='l1l2',  V= <br>
%             argmin 0.5||U-V||_F^2 + lambda||V||_{1/2}<br>
%         if param.regul='l1linf',  V= <br>
%             argmin 0.5||U-V||_F^2 + lambda||V||_{1/inf}<br>
%         if param.regul='l1l2+l1',  V= <br>
%             argmin 0.5||U-V||_F^2 + lambda||V||_{1/2} + lambda_2||V||_{1/1}<br>
%         if param.regul='l1linf+l1',  V= <br>
%             argmin 0.5||U-V||_F^2 + lambda||V||_{1/inf} + lambda_2||V||_{1/1}<br>
%         if param.regul='l1linf+row-column',  V= <br>
%             argmin 0.5||U-V||_F^2 + lambda||V||_{1/inf} + lambda_2||V'||_{1/inf}<br>
%         if param.regul='trace-norm',  V= <br>
%             argmin 0.5||U-V||_F^2 + lambda||V||_*<br>
%         if param.regul='rank',  V= <br>
%             argmin 0.5||U-V||_F^2 + lambda rank(V)<br>
%         if param.regul='none',  V= <br>
%             argmin 0.5||U-V||_F^2 <br>
%         <br>
%         for all these regularizations, it is possible to enforce non-negativity constraints<br>
%         with the option param.pos, and to prevent the last row of U to be regularized, with<br>
%         the option param.intercept<br>
%<br>
% Inputs: U:  double m x n matrix   (input signals)<br>
%               m is the signal size<br>
%         param: struct<br>
%               param.lambda  (regularization parameter)<br>
%               param.regul (choice of regularization, see above)<br>
%               param.lambda2  (optional, regularization parameter)<br>
%               param.lambda3  (optional, regularization parameter)<br>
%               param.verbose (optional, verbosity level, false by default)<br>
%               param.intercept (optional, last row of U is not regularized,<br>
%                 false by default)<br>
%               param.transpose (optional, transpose the matrix in the regularization function)<br>
%               param.size_group (optional, for regularization functions assuming a group<br>
%                 structure). It is a scalar. When param.groups is not specified, it assumes<br>
%                 that the groups are the sets of consecutive elements of size param.size_group<br>
%               param.groups (int32, optional, for regularization functions assuming a group<br>
%                 structure. It is an int32 vector of size m containing the group indices of the<br>
%                 variables (first group is 1).<br>
%               param.pos (optional, adds positivity constraints on the<br>
%                 coefficients, false by default)<br>
%               param.numThreads (optional, number of threads for exploiting<br>
%                 multi-core / multi-cpus. By default, it takes the value -1,<br>
%                 which automatically selects all the available CPUs/cores).<br>
%<br>
% Output: V: double m x n matrix (output coefficients)<br>
%         val_regularizer: double 1 x n vector (value of the regularization<br>
%         term at the optimum).<br>
%<br>
% Author: Julien Mairal, 2010</span></td></tr>
</table><p>
The following piece of code illustrates how to use this function.
</p><table class="lstframe c011"><tr><td class="mouselstlisting">U=<span class="c002">randn</span>(100,1000);<br>
<br>
param.lambda=0.1; <span class="c001">% regularization parameter</span><br>
param.num_threads=-1; <span class="c001">% all cores (-1 by default)</span><br>
param.verbose=true;   <span class="c001">% verbosity, false by default<br>
<br>
% test l0</span><br>
<span class="c002">fprintf</span>(<span class="c003">'\nprox l0\n'</span>);<br>
param.regul=<span class="c003">'l0'</span>;<br>
param.pos=false;       <span class="c001">% false by default</span><br>
param.intercept=false; <span class="c001">% false by default</span><br>
alpha=mexProximalFlat(U,param);<br>
<br>
<span class="c001">% test l1</span><br>
<span class="c002">fprintf</span>(<span class="c003">'\nprox l1, intercept, positivity constraint\n'</span>);<br>
param.regul=<span class="c003">'l1'</span>;<br>
param.pos=true;       <span class="c001">% can be used with all the other regularizations</span><br>
param.intercept=true; <span class="c001">% can be used with all the other regularizations     </span><br>
alpha=mexProximalFlat(U,param);<br>
<br>
<span class="c001">% test l2</span><br>
<span class="c002">fprintf</span>(<span class="c003">'\nprox squared-l2\n'</span>);<br>
param.regul=<span class="c003">'l2'</span>;<br>
param.pos=false;<br>
param.intercept=false;<br>
alpha=mexProximalFlat(U,param);<br>
<br>
<span class="c001">% test elastic-net</span><br>
<span class="c002">fprintf</span>(<span class="c003">'\nprox elastic-net\n'</span>);<br>
param.regul=<span class="c003">'elastic-net'</span>;<br>
param.lambda2=0.1;<br>
alpha=mexProximalFlat(U,param);<br>
<br>
<span class="c001">% test fused-lasso</span><br>
<span class="c002">fprintf</span>(<span class="c003">'\nprox fused lasso\n'</span>);<br>
param.regul=<span class="c003">'fused-lasso'</span>;<br>
param.lambda2=0.1;<br>
param.lambda3=0.1;<br>
alpha=mexProximalFlat(U,param);<br>
<br>
<span class="c001">% test l1l2</span><br>
<span class="c002">fprintf</span>(<span class="c003">'\nprox mixed norm l1/l2\n'</span>);<br>
param.regul=<span class="c003">'l1l2'</span>;<br>
alpha=mexProximalFlat(U,param);<br>
<br>
<span class="c001">% test l1linf</span><br>
<span class="c002">fprintf</span>(<span class="c003">'\nprox mixed norm l1/linf\n'</span>);<br>
param.regul=<span class="c003">'l1linf'</span>;<br>
alpha=mexProximalFlat(U,param);<br>
<br>
<span class="c001">% test l1l2+l1</span><br>
<span class="c002">fprintf</span>(<span class="c003">'\nprox mixed norm l1/l2 + l1\n'</span>);<br>
param.regul=<span class="c003">'l1l2+l1'</span>;<br>
param.lambda2=0.1;<br>
alpha=mexProximalFlat(U,param);<br>
<br>
<span class="c001">% test l1linf+l1</span><br>
<span class="c002">fprintf</span>(<span class="c003">'\nprox mixed norm l1/linf + l1\n'</span>);<br>
param.regul=<span class="c003">'l1linf+l1'</span>;<br>
param.lambda2=0.1;<br>
alpha=mexProximalFlat(U,param);<br>
<br>
<span class="c001">% test l1linf-row-column</span><br>
<span class="c002">fprintf</span>(<span class="c003">'\nprox mixed norm l1/linf on rows and columns\n'</span>);<br>
param.regul=<span class="c003">'l1linf-row-column'</span>;<br>
param.lambda2=0.1;<br>
alpha=mexProximalFlat(U,param);<br>
<br>
<span class="c001">% test none</span><br>
<span class="c002">fprintf</span>(<span class="c003">'\nprox no regularization\n'</span>);<br>
param.regul=<span class="c003">'none'</span>;<br>
alpha=mexProximalFlat(U,param);</td></tr>
</table>
<!--TOC subsection id="sec26" Function mexProximalTree-->
<h3 class="subsection" id="sec26">5.3  Function mexProximalTree</h3><!--SEC END --><p>
This function computes the proximal operators associated to tree-structured regularization functions, for input signals <span class="c009">U</span>=[<span class="c009">u</span><sup>1</sup>,…,<span class="c009">u</span><sup><span class="c007">n</span></sup>] in ℝ<sup><span class="c007">p</span> × <span class="c007">n</span></sup>, and a tree-structured set of groups [<a href="#jenatton3">15</a>], it computes a matrix <span class="c009">V</span>=[<span class="c009">v</span><sup>1</sup>,…,<span class="c009">v</span><sup><span class="c007">n</span></sup>] in ℝ<sup><span class="c007">p</span> × <span class="c007">n</span></sup>. When one uses a regularization function on vectors, it computes a column <span class="c009">v</span> of <span class="c009">V</span> for every column <span class="c009">u</span> of <span class="c009">U</span>:
</p><table class="display dcenter"><tr class="c016"><td class="dcell">
</td><td class="dcell"><table class="display"><tr><td class="dcell c012">&nbsp;</td></tr>
<tr><td class="dcell c012">min</td></tr>
<tr><td class="dcell c012"><span class="c009">v</span> ∈ ℝ<sup><span class="c007">p</span></sup></td></tr>
</table></td><td class="dcell"> </td><td class="dcell"><table class="display"><tr><td class="dcell c012">1</td></tr>
<tr><td class="hbar"></td></tr>
<tr><td class="dcell c012">2</td></tr>
</table></td><td class="dcell">||<span class="c009">u</span>−<span class="c009">v</span>||<sub>2</sub><sup>2</sup> + λ </td><td class="dcell"><table class="display"><tr><td class="dcell c012">&nbsp;</td></tr>
<tr><td class="dcell c012"><span class="c006">∑</span></td></tr>
<tr><td class="dcell c012"><span class="c007">g</span> ∈ <span class="c008">T</span></td></tr>
</table></td><td class="dcell"> η<sup><span class="c007">g</span></sup> ||<span class="c009">v</span><sub><span class="c007">g</span></sub>||<sub>2</sub>,
    (40)</td></tr>
</table><p>
or
</p><table class="display dcenter"><tr class="c016"><td class="dcell">
</td><td class="dcell"><table class="display"><tr><td class="dcell c012">&nbsp;</td></tr>
<tr><td class="dcell c012">min</td></tr>
<tr><td class="dcell c012"><span class="c009">v</span> ∈ ℝ<sup><span class="c007">p</span></sup></td></tr>
</table></td><td class="dcell"> </td><td class="dcell"><table class="display"><tr><td class="dcell c012">1</td></tr>
<tr><td class="hbar"></td></tr>
<tr><td class="dcell c012">2</td></tr>
</table></td><td class="dcell">||<span class="c009">u</span>−<span class="c009">v</span>||<sub>2</sub><sup>2</sup> + λ </td><td class="dcell"><table class="display"><tr><td class="dcell c012">&nbsp;</td></tr>
<tr><td class="dcell c012"><span class="c006">∑</span></td></tr>
<tr><td class="dcell c012"><span class="c007">g</span> ∈ <span class="c008">T</span></td></tr>
</table></td><td class="dcell"> η<sup><span class="c007">g</span></sup> ||<span class="c009">v</span><sub><span class="c007">g</span></sub>||<sub>∞</sub>,
    (41)</td></tr>
</table><p>
or
</p><table class="display dcenter"><tr class="c016"><td class="dcell">
</td><td class="dcell"><table class="display"><tr><td class="dcell c012">&nbsp;</td></tr>
<tr><td class="dcell c012">min</td></tr>
<tr><td class="dcell c012"><span class="c009">v</span> ∈ ℝ<sup><span class="c007">p</span></sup></td></tr>
</table></td><td class="dcell"> </td><td class="dcell"><table class="display"><tr><td class="dcell c012">1</td></tr>
<tr><td class="hbar"></td></tr>
<tr><td class="dcell c012">2</td></tr>
</table></td><td class="dcell">||<span class="c009">u</span>−<span class="c009">v</span>||<sub>2</sub><sup>2</sup> + λ </td><td class="dcell"><table class="display"><tr><td class="dcell c012">&nbsp;</td></tr>
<tr><td class="dcell c012"><span class="c006">∑</span></td></tr>
<tr><td class="dcell c012"><span class="c007">g</span> ∈ <span class="c008">T</span></td></tr>
</table></td><td class="dcell"> δ<sup><span class="c007">g</span></sup>(<span class="c009">v</span>),
    (42)</td></tr>
</table><p>
where δ<sup><span class="c007">g</span></sup>(<span class="c009">v</span>)=0 if <span class="c009">v</span><sub><span class="c007">g</span></sub>=0 and 1 otherwise (see appendix of [<a href="#jenatton4">16</a>]).</p><p>When the multi-task tree-structured regularization function is used, it solves
</p><table class="display dcenter"><tr class="c016"><td class="dcell">
</td><td class="dcell"><table class="display"><tr><td class="dcell c012">&nbsp;</td></tr>
<tr><td class="dcell c012">min</td></tr>
<tr><td class="dcell c012"><span class="c009">V</span> ∈ ℝ<sup><span class="c007">p</span>× <span class="c007">n</span></sup></td></tr>
</table></td><td class="dcell"> </td><td class="dcell"><table class="display"><tr><td class="dcell c012">1</td></tr>
<tr><td class="hbar"></td></tr>
<tr><td class="dcell c012">2</td></tr>
</table></td><td class="dcell">||<span class="c009">U</span>−<span class="c009">V</span>||<sub><span class="c007">F</span></sub><sup>2</sup> + λ </td><td class="dcell"><table class="display"><tr><td class="dcell c012"><span class="c007">n</span></td></tr>
<tr><td class="dcell c012"><span class="c006">∑</span></td></tr>
<tr><td class="dcell c012"><span class="c007">i</span>=1</td></tr>
</table></td><td class="dcell"><table class="display"><tr><td class="dcell c012">&nbsp;</td></tr>
<tr><td class="dcell c012"><span class="c006">∑</span></td></tr>
<tr><td class="dcell c012"><span class="c007">g</span> ∈ <span class="c008">T</span></td></tr>
</table></td><td class="dcell"> η<sup><span class="c007">g</span></sup> ||<span class="c009">v</span><sub><span class="c007">g</span></sub><sup><span class="c007">i</span></sup>||<sub>∞</sub>+ λ<sub>2</sub></td><td class="dcell"><table class="display"><tr><td class="dcell c012">&nbsp;</td></tr>
<tr><td class="dcell c012"><span class="c006">∑</span></td></tr>
<tr><td class="dcell c012"><span class="c007">g</span> ∈ <span class="c008">T</span></td></tr>
</table></td><td class="dcell"> </td><td class="dcell"><table class="display"><tr><td class="dcell c012">&nbsp;</td></tr>
<tr><td class="dcell c012">max</td></tr>
<tr><td class="dcell c012"><span class="c007">j</span> ∈ <span class="c007">g</span></td></tr>
</table></td><td class="dcell"> ||<span class="c009">v</span><sub><span class="c007">g</span></sub><sup><span class="c007">j</span></sup>||<sub>∞</sub>,
    (43)</td></tr>
</table><p> 
which is a formulation presented in [<a href="#mairal10">22</a>].</p><p>This function can also be used for computing the proximal operators addressed by mexProximalFlat (it will just not take into account the tree structure). The way the tree is incoded is presented below, (and examples are given in the file test_ProximalTree.m, with more usage details:
</p><table class="lstframe c011"><tr><td class="mouselstlisting"><span class="c001">% <br>
% Usage:  [V [val_regularizer]]=mexProximalTree(U,tree,param);<br>
%<br>
% Name: mexProximalTree<br>
%<br>
% Description: mexProximalTree computes a proximal operator. Depending<br>
%         on the value of param.regul, it computes <br>
%<br>
%         Given an input matrix U=[u^1,\ldots,u^n], and a tree-structured set of groups T,<br>
%         it returns a matrix V=[v^1,\ldots,v^n]:<br>
%         <br>
%         when the regularization function is for vectors,<br>
%         for every column u of U, it compute a column v of V solving<br>
%         if param.regul='tree-l0'<br>
%             argmin 0.5||u-v||_2^2 + lambda \sum_{g \in T} \delta^g(v)<br>
%         if param.regul='tree-l2'<br>
%           for all i, v^i = <br>
%             argmin 0.5||u-v||_2^2 + lambda\sum_{g \in T} \eta_g||v_g||_2<br>
%         if param.regul='tree-linf'<br>
%           for all i, v^i = <br>
%             argmin 0.5||u-v||_2^2 + lambda\sum_{g \in T} \eta_g||v_g||_inf<br>
%<br>
%         when the regularization function is for matrices:<br>
%         if param.regul='multi-task-tree'<br>
%            V=argmin 0.5||U-V||_F^2 + lambda \sum_{i=1}^n\sum_{g \in T} \eta_g||v^i_g||_inf + ...<br>
%                                                lambda_2 \sum_{g \in T} \eta_g max_{j in g}||V_j||_{inf}<br>
%         <br>
%         it can also be used with any non-tree-structured regularization addressed by mexProximalFlat<br>
%<br>
%         for all these regularizations, it is possible to enforce non-negativity constraints<br>
%         with the option param.pos, and to prevent the last row of U to be regularized, with<br>
%         the option param.intercept<br>
%<br>
% Inputs: U:  double m x n matrix   (input signals)<br>
%               m is the signal size<br>
%         tree: struct<br>
%               with four fields, eta_g, groups, own_variables and N_own_variables.<br>
%<br>
%               The tree structure requires a particular organization of groups and variables<br>
%                  * Let us denote by N = |T|, the number of groups.<br>
%                    the groups should be ordered T={g1,g2,\ldots,gN} such that if gi is included<br>
%                    in gj, then j &lt;= i. g1 should be the group at the root of the tree <br>
%                    and contains every variable.<br>
%                  * Every group is a set of  contiguous indices for instance <br>
%                    gi={3,4,5} or gi={4,5,6,7} or gi={4}, but not {3,5};<br>
%                  * We define root(gi) as the indices of the variables that are in gi,<br>
%                    but not in its descendants. For instance for<br>
%                    T={ g1={1,2,3,4},g2={2,3},g3={4} }, then, root(g1)={1}, <br>
%                    root(g2)={2,3}, root(g3)={4},<br>
%                    We assume that for all i, root(gi) is a set of contigous variables<br>
%                  * We assume that the smallest of root(gi) is also the smallest index of gi.<br>
%<br>
%                  For instance, <br>
%                    T={ g1={1,2,3,4},g2={2,3},g3={4} }, is a valid set of groups.<br>
%                    but we can not have<br>
%                    T={ g1={1,2,3,4},g2={1,2},g3={3} }, since root(g1)={4} and 4 is not the<br>
%                    smallest element in g1.<br>
%<br>
%               We do not lose generality with these assumptions since they can be fullfilled for any<br>
%               tree-structured set of groups after a permutation of variables and a correct ordering of the<br>
%               groups.<br>
%               see more examples in test_ProximalTree.m of valid tree-structured sets of groups.<br>
%               <br>
%               The first fields sets the weights for every group<br>
%                  tree.eta_g            double N vector <br>
%  <br>
%               The next field sets inclusion relations between groups <br>
%               (but not between groups and variables):<br>
%                  tree.groups           sparse (double or boolean) N x N matrix  <br>
%                  the (i,j) entry is non-zero if and only if i is different than j and <br>
%                  gi is included in gj.<br>
%                  the first column corresponds to the group at the root of the tree.<br>
%<br>
%               The next field define the smallest index of each group gi, <br>
%               which is also the smallest index of root(gi)<br>
%               tree.own_variables    int32 N vector<br>
%<br>
%               The next field define for each group gi, the size of root(gi)<br>
%               tree.N_own_variables  int32 N vector <br>
%<br>
%               examples are given in test_ProximalTree.m<br>
%<br>
%         param: struct<br>
%               param.lambda  (regularization parameter)<br>
%               param.regul (choice of regularization, see above)<br>
%               param.lambda2  (optional, regularization parameter)<br>
%               param.lambda3  (optional, regularization parameter)<br>
%               param.verbose (optional, verbosity level, false by default)<br>
%               param.intercept (optional, last row of U is not regularized,<br>
%                 false by default)<br>
%               param.pos (optional, adds positivity constraints on the<br>
%                 coefficients, false by default)<br>
%               param.transpose (optional, transpose the matrix in the regularization function)<br>
%               param.size_group (optional, for regularization functions assuming a group<br>
%                 structure). It is a scalar. When param.groups is not specified, it assumes<br>
%                 that the groups are the sets of consecutive elements of size param.size_group<br>
%               param.numThreads (optional, number of threads for exploiting<br>
%                 multi-core / multi-cpus. By default, it takes the value -1,<br>
%                 which automatically selects all the available CPUs/cores).<br>
%<br>
% Output: V: double m x n matrix (output coefficients)<br>
%         val_regularizer: double 1 x n vector (value of the regularization<br>
%         term at the optimum).<br>
%<br>
%<br>
% Author: Julien Mairal, 2010</span></td></tr>
</table><p>
The following piece of code illustrates how to use this function.
</p><table class="lstframe c011"><tr><td class="mouselstlisting">U=<span class="c002">randn</span>(10,1000);<br>
<br>
param.lambda=0.1; <span class="c001">% regularization parameter</span><br>
param.num_threads=-1; <span class="c001">% all cores (-1 by default)</span><br>
param.verbose=true;   <span class="c001">% verbosity, false by default</span><br>
param.pos=false;       <span class="c001">% can be used with all the other regularizations</span><br>
param.intercept=false; <span class="c001">% can be used with all the other regularizations     </span><br>
<br>
<span class="c002">fprintf</span>(<span class="c003">'First tree example\n'</span>);<br>
<span class="c001">% Example 1 of tree structure<br>
% tree structured groups:<br>
% g1= {0 1 2 3 4 5 6 7 8 9}<br>
% g2= {2 3 4}<br>
% g3= {5 6 7 8 9}</span><br>
tree.own_variables=int32([0 2 5]);   <span class="c001">% pointer to the first variable of each group</span><br>
tree.N_own_variables=int32([2 3 5]); <span class="c001">% number of "root" variables in each group</span><br>
                              <span class="c001">% (variables that are in a group, but not in its descendants).</span><br>
                              <span class="c001">% for instance root(g1)={0,1}, root(g2)={2 3 4}, root(g3)={5 6 7 8 9}</span><br>
tree.eta_g=[1 1 1];           <span class="c001">% weights for each group, they should be non-zero to use fenchel duality</span><br>
tree.groups=<span class="c002">sparse</span>([0 0 0; ...<br>
                    1 0 0; ...<br>
                    1 0 0]);    <span class="c001">% first group should always be the root of the tree</span><br>
                                <span class="c001">% non-zero entriees mean inclusion relation ship, here g2 is a children of g1,</span><br>
                                <span class="c001">% g3 is a children of g1</span><br>
<br>
<span class="c002">fprintf</span>(<span class="c003">'\ntest prox tree-l0\n'</span>);<br>
param.regul=<span class="c003">'tree-l0'</span>;<br>
alpha=mexProximalTree(U,tree,param);<br>
<br>
<span class="c002">fprintf</span>(<span class="c003">'\ntest prox tree-l2\n'</span>);<br>
param.regul=<span class="c003">'tree-l2'</span>;<br>
alpha=mexProximalTree(U,tree,param);<br>
<br>
<span class="c002">fprintf</span>(<span class="c003">'\ntest prox tree-linf\n'</span>);<br>
param.regul=<span class="c003">'tree-linf'</span>;<br>
alpha=mexProximalTree(U,tree,param);<br>
<br>
<span class="c002">fprintf</span>(<span class="c003">'Second tree example\n'</span>);<br>
<span class="c001">% Example 2 of tree structure<br>
% tree structured groups:<br>
% g1= {0 1 2 3 4 5 6 7 8 9}    root(g1) = { };<br>
% g2= {0 1 2 3 4 5}            root(g2) = {0 1 2};<br>
% g3= {3 4}                    root(g3) = {3 4};<br>
% g4= {5}                      root(g4) = {5};<br>
% g5= {6 7 8 9}                root(g5) = { };<br>
% g6= {6 7}                    root(g6) = {6 7};<br>
% g7= {8 9}                    root(g7) = {8};<br>
% g8 = {9}                     root(g8) = {9};</span><br>
tree.own_variables=  int32([0 0 3 5 6 6 8 9]);   <span class="c001">% pointer to the first variable of each group</span><br>
tree.N_own_variables=int32([0 3 2 1 0 2 1 1]); <span class="c001">% number of "root" variables in each group</span><br>
tree.eta_g=[1 1 1 2 2 2 2.5 2.5];<br>
tree.groups=<span class="c002">sparse</span>([0 0 0 0 0 0 0 0; ...<br>
                    1 0 0 0 0 0 0 0; ...<br>
                    0 1 0 0 0 0 0 0; ...<br>
                    0 1 0 0 0 0 0 0; ...<br>
                    1 0 0 0 0 0 0 0; ...<br>
                    0 0 0 0 1 0 0 0; ...<br>
                    0 0 0 0 1 0 0 0; ...<br>
                    0 0 0 0 0 0 1 0]);  <span class="c001">% first group should always be the root of the tree</span><br>
<br>
<span class="c002">fprintf</span>(<span class="c003">'\ntest prox tree-l0\n'</span>);<br>
param.regul=<span class="c003">'tree-l0'</span>;<br>
alpha=mexProximalTree(U,tree,param);<br>
<br>
<span class="c002">fprintf</span>(<span class="c003">'\ntest prox tree-l2\n'</span>);<br>
param.regul=<span class="c003">'tree-l2'</span>;<br>
alpha=mexProximalTree(U,tree,param);<br>
<br>
<span class="c002">fprintf</span>(<span class="c003">'\ntest prox tree-linf\n'</span>);<br>
param.regul=<span class="c003">'tree-linf'</span>;<br>
alpha=mexProximalTree(U,tree,param);<br>
<br>
<span class="c001">% mexProximalTree also works with non-tree-structured regularization functions</span><br>
<span class="c002">fprintf</span>(<span class="c003">'\nprox l1, intercept, positivity constraint\n'</span>);<br>
param.regul=<span class="c003">'l1'</span>;<br>
param.pos=true;       <span class="c001">% can be used with all the other regularizations</span><br>
param.intercept=true; <span class="c001">% can be used with all the other regularizations     </span><br>
alpha=mexProximalTree([U; ones(1,<span class="c002">size</span>(U,2))],tree,param);<br>
<br>
<span class="c001">% Example of multi-task tree</span><br>
<span class="c002">fprintf</span>(<span class="c003">'\nprox multi-task tree\n'</span>);<br>
param.pos=false;<br>
param.intercept=false;<br>
param.lambda2=param.lambda;<br>
param.regul=<span class="c003">'multi-task-tree'</span>;  <span class="c001">% with linf</span><br>
alpha=mexProximalTree(U,tree,param);<br>
<br>
<br>
tree.own_variables=int32([0 1 2 3 4 5 6]);   <span class="c001">% pointer to the first variable of each group</span><br>
tree.N_own_variables=int32([1 1 1 1 1 1]); <span class="c001">% number of "root" variables in each group</span><br>
                              <span class="c001">% (variables that are in a group, but not in its descendants).</span><br>
                              <span class="c001">% for instance root(g1)={0,1}, root(g2)={2 3 4}, root(g3)={5 6 7 8 9}</span><br>
tree.eta_g=[1 1 1 1 1 1];           <span class="c001">% weights for each group, they should be non-zero to use fenchel duality</span><br>
tree.groups=<span class="c002">sparse</span>([0 0 0; ...<br>
                    1 0 0; ...<br>
                    1 0 0]);    <span class="c001">% first group should always be the root of the tree</span><br>
                                <span class="c001">% non-zero entriees mean inclusion relation ship, here g2 is a children of g1,</span><br>
                                <span class="c001">% g3 is a children of g1</span></td></tr>
</table>
<!--TOC subsection id="sec27" Function mexProximalGraph-->
<h3 class="subsection" id="sec27">5.4  Function mexProximalGraph</h3><!--SEC END --><p>
This function computes the proximal operators associated to structured sparse regularization, for input signals <span class="c009">U</span>=[<span class="c009">u</span><sup>1</sup>,…,<span class="c009">u</span><sup><span class="c007">n</span></sup>] in ℝ<sup><span class="c007">p</span> × <span class="c007">n</span></sup>, and a set of groups [<a href="#mairal10">22</a>], it returns a matrix <span class="c009">V</span>=[<span class="c009">v</span><sup>1</sup>,…,<span class="c009">v</span><sup><span class="c007">n</span></sup>] in ℝ<sup><span class="c007">p</span> × <span class="c007">n</span></sup>.
When one uses a regularization function on vectors, it computes a column <span class="c009">v</span> of <span class="c009">V</span> for every column <span class="c009">u</span> of <span class="c009">U</span>:
</p><table class="display dcenter"><tr class="c016"><td class="dcell">
</td><td class="dcell"><table class="display"><tr><td class="dcell c012">&nbsp;</td></tr>
<tr><td class="dcell c012">min</td></tr>
<tr><td class="dcell c012"><span class="c009">v</span> ∈ ℝ<sup><span class="c007">p</span></sup></td></tr>
</table></td><td class="dcell"> </td><td class="dcell"><table class="display"><tr><td class="dcell c012">1</td></tr>
<tr><td class="hbar"></td></tr>
<tr><td class="dcell c012">2</td></tr>
</table></td><td class="dcell">||<span class="c009">u</span>−<span class="c009">v</span>||<sub>2</sub><sup>2</sup> + λ </td><td class="dcell"><table class="display"><tr><td class="dcell c012">&nbsp;</td></tr>
<tr><td class="dcell c012"><span class="c006">∑</span></td></tr>
<tr><td class="dcell c012"><span class="c007">g</span> ∈ <span class="c008">G</span></td></tr>
</table></td><td class="dcell"> η<sup><span class="c007">g</span></sup> ||<span class="c009">v</span><sub><span class="c007">g</span></sub>||<sub>∞</sub>,
    (44)</td></tr>
</table><p>
or with a regularization function on matrices, it computes <span class="c009">V</span> solving
</p><table class="display dcenter"><tr class="c016"><td class="dcell">
</td><td class="dcell"><table class="display"><tr><td class="dcell c012">&nbsp;</td></tr>
<tr><td class="dcell c012">min</td></tr>
<tr><td class="dcell c012"><span class="c009">V</span> ∈ ℝ<sup><span class="c007">p</span>× <span class="c007">n</span></sup></td></tr>
</table></td><td class="dcell"> </td><td class="dcell"><table class="display"><tr><td class="dcell c012">1</td></tr>
<tr><td class="hbar"></td></tr>
<tr><td class="dcell c012">2</td></tr>
</table></td><td class="dcell">||<span class="c009">U</span>−<span class="c009">V</span>||<sub><span class="c007">F</span></sub><sup>2</sup> + λ </td><td class="dcell"><table class="display"><tr><td class="dcell c012"><span class="c007">n</span></td></tr>
<tr><td class="dcell c012"><span class="c006">∑</span></td></tr>
<tr><td class="dcell c012"><span class="c007">i</span>=1</td></tr>
</table></td><td class="dcell"><table class="display"><tr><td class="dcell c012">&nbsp;</td></tr>
<tr><td class="dcell c012"><span class="c006">∑</span></td></tr>
<tr><td class="dcell c012"><span class="c007">g</span> ∈ <span class="c008">G</span></td></tr>
</table></td><td class="dcell"> η<sup><span class="c007">g</span></sup> ||<span class="c009">v</span><sub><span class="c007">g</span></sub><sup><span class="c007">i</span></sup>||<sub>∞</sub>+ λ<sub>2</sub></td><td class="dcell"><table class="display"><tr><td class="dcell c012">&nbsp;</td></tr>
<tr><td class="dcell c012"><span class="c006">∑</span></td></tr>
<tr><td class="dcell c012"><span class="c007">g</span> ∈ <span class="c008">G</span></td></tr>
</table></td><td class="dcell"> </td><td class="dcell"><table class="display"><tr><td class="dcell c012">&nbsp;</td></tr>
<tr><td class="dcell c012">max</td></tr>
<tr><td class="dcell c012"><span class="c007">j</span> ∈ <span class="c007">g</span></td></tr>
</table></td><td class="dcell"> ||<span class="c009">v</span><sub><span class="c007">g</span></sub><sup><span class="c007">j</span></sup>||<sub>∞</sub>,
    (45)</td></tr>
</table><p> 
This function can also be used for computing the proximal operators addressed by mexProximalFlat. The way the graph is incoded is presented below (and also in the example file test_ProximalGraph.m, with more usage details:
</p><table class="lstframe c011"><tr><td class="mouselstlisting"><span class="c001">% <br>
% Usage:   [V [val_regularizer]]=mexProximalGraph(U,graph,param);<br>
%<br>
% Name: mexProximalGraph<br>
%<br>
% Description: mexProximalGraph computes a proximal operator. Depending<br>
%         on the value of param.regul, it computes <br>
%<br>
%         Given an input matrix U=[u^1,\ldots,u^n], and a set of groups G,<br>
%         it computes a matrix V=[v^1,\ldots,v^n] such that<br>
%<br>
%         if param.regul='graph'<br>
%         for every column u of U, it computes a column v of V solving<br>
%             argmin 0.5||u-v||_2^2 + lambda\sum_{g \in G} \eta_g||v_g||_inf<br>
%<br>
%         if param.regul='graph+ridge'<br>
%         for every column u of U, it computes a column v of V solving<br>
%             argmin 0.5||u-v||_2^2 + lambda\sum_{g \in G} \eta_g||v_g||_inf + lambda_2||v||_2^2<br>
%<br>
%<br>
%         if param.regul='multi-task-graph'<br>
%            V=argmin 0.5||U-V||_F^2 + lambda \sum_{i=1}^n\sum_{g \in G} \eta_g||v^i_g||_inf + ...<br>
%                                                lambda_2 \sum_{g \in G} \eta_g max_{j in g}||V_j||_{inf}<br>
%         <br>
%         it can also be used with any regularization addressed by mexProximalFlat<br>
%<br>
%         for all these regularizations, it is possible to enforce non-negativity constraints<br>
%         with the option param.pos, and to prevent the last row of U to be regularized, with<br>
%         the option param.intercept<br>
%<br>
% Inputs: U:  double p x n matrix   (input signals)<br>
%               m is the signal size<br>
%         graph: struct<br>
%               with three fields, eta_g, groups, and groups_var<br>
%<br>
%               The first fields sets the weights for every group<br>
%                  graph.eta_g            double N vector <br>
%  <br>
%               The next field sets inclusion relations between groups <br>
%               (but not between groups and variables):<br>
%                  graph.groups           sparse (double or boolean) N x N matrix  <br>
%                  the (i,j) entry is non-zero if and only if i is different than j and <br>
%                  gi is included in gj.<br>
%               <br>
%               The next field sets inclusion relations between groups and variables<br>
%                  graph.groups_var       sparse (double or boolean) p x N matrix<br>
%                  the (i,j) entry is non-zero if and only if the variable i is included <br>
%                  in gj, but not in any children of gj.<br>
%<br>
%               examples are given in test_ProximalGraph.m<br>
%<br>
%         param: struct<br>
%               param.lambda  (regularization parameter)<br>
%               param.regul (choice of regularization, see above)<br>
%               param.lambda2  (optional, regularization parameter)<br>
%               param.lambda3  (optional, regularization parameter)<br>
%               param.verbose (optional, verbosity level, false by default)<br>
%               param.intercept (optional, last row of U is not regularized,<br>
%                 false by default)<br>
%               param.pos (optional, adds positivity constraints on the<br>
%                 coefficients, false by default)<br>
%               param.numThreads (optional, number of threads for exploiting<br>
%                 multi-core / multi-cpus. By default, it takes the value -1,<br>
%                 which automatically selects all the available CPUs/cores).<br>
%<br>
% Output: V: double p x n matrix (output coefficients)<br>
%         val_regularizer: double 1 x n vector (value of the regularization<br>
%         term at the optimum).<br>
%<br>
% Author: Julien Mairal, 2010</span></td></tr>
</table><p>
The following piece of code illustrates how to use this function.
</p><table class="lstframe c011"><tr><td class="mouselstlisting">U=<span class="c002">randn</span>(10,1000);<br>
<br>
param.lambda=0.1; <span class="c001">% regularization parameter</span><br>
param.num_threads=-1; <span class="c001">% all cores (-1 by default)</span><br>
param.verbose=true;   <span class="c001">% verbosity, false by default</span><br>
param.pos=false;       <span class="c001">% can be used with all the other regularizations</span><br>
param.intercept=false; <span class="c001">% can be used with all the other regularizations     </span><br>
<br>
<span class="c002">fprintf</span>(<span class="c003">'First graph example\n'</span>);<br>
<span class="c001">% Example 1 of graph structure<br>
% groups:<br>
% g1= {0 1 2 3}<br>
% g2= {3 4 5 6}<br>
% g3= {6 7 8 9}</span><br>
graph.eta_g=[1 1 1];<br>
graph.groups=<span class="c002">sparse</span>(<span class="c002">zeros</span>(3));<br>
graph.groups_var=<span class="c002">sparse</span>([1 0 0;<br>
                         1 0 0;<br>
                         1 0 0;<br>
                         1 1 0;<br>
                         0 1 0;<br>
                         0 1 0;<br>
                         0 1 1;<br>
                         0 0 1;<br>
                         0 0 1;<br>
                         0 0 1]);<br>
<br>
<span class="c002">fprintf</span>(<span class="c003">'\ntest prox graph\n'</span>);<br>
param.regul=<span class="c003">'graph'</span>;<br>
alpha=mexProximalGraph(U,graph,param);<br>
<br>
<span class="c001">% Example 2 of graph structure<br>
% groups:<br>
% g1= {0 1 2 3}<br>
% g2= {3 4 5 6}<br>
% g3= {6 7 8 9}<br>
% g4= {0 1 2 3 4 5}<br>
% g5= {6 7 8}</span><br>
graph.eta_g=[1 1 1 1 1];<br>
graph.groups=<span class="c002">sparse</span>([0 0 0 1 0;<br>
                     0 0 0 0 0;<br>
                     0 0 0 0 0;<br>
                     0 0 0 0 0;<br>
                     0 0 1 0 0]);   <span class="c001">% g5 is included in g3, and g2 is included in g4</span><br>
graph.groups_var=<span class="c002">sparse</span>([1 0 0 0 0;<br>
                         1 0 0 0 0;<br>
                         1 0 0 0 0 ;<br>
                         1 1 0 0 0;<br>
                         0 1 0 1 0;<br>
                         0 1 0 1 0;<br>
                         0 1 0 0 1;<br>
                         0 0 0 0 1;<br>
                         0 0 0 0 1;<br>
                         0 0 1 0 0]); <span class="c001">% represents direct inclusion relations </span><br>
                                      <span class="c001">% between groups (columns) and variables (rows)</span><br>
<br>
<span class="c002">fprintf</span>(<span class="c003">'\ntest prox graph\n'</span>);<br>
param.regul=<span class="c003">'graph'</span>;<br>
alpha=mexProximalGraph(U,graph,param);<br>
<br>
<span class="c002">fprintf</span>(<span class="c003">'\ntest prox multi-task-graph\n'</span>);<br>
param.regul=<span class="c003">'multi-task-graph'</span>;<br>
param.lambda2=0.1;<br>
alpha=mexProximalGraph(U,graph,param);<br>
<br>
<span class="c002">fprintf</span>(<span class="c003">'\ntest no regularization\n'</span>);<br>
param.regul=<span class="c003">'none'</span>;<br>
alpha=mexProximalGraph(U,graph,param);</td></tr>
</table>
<!--TOC subsection id="sec28" Function mexProximalPathCoding-->
<h3 class="subsection" id="sec28">5.5  Function mexProximalPathCoding</h3><!--SEC END --><p>
This function computes the proximal operators associated to the path coding penalties of [<a href="#mairal14">24</a>]. 
</p><table class="lstframe c011"><tr><td class="mouselstlisting"><span class="c001">% <br>
% Usage:   [V [val_regularizer]]=mexProximalPathCoding(U,DAG,param);<br>
%<br>
% Name: mexProximalPathCoding<br>
%<br>
% Description: mexProximalPathCoding computes a proximal operator for<br>
%         the path coding penalties of http://arxiv.org/abs/1204.4539<br>
%<br>
%         Given an input matrix U=[u^1,\ldots,u^n], <br>
%<br>
%<br>
% Inputs: U:  double p x n matrix   (input signals)<br>
%               m is the signal size<br>
%         DAG:  struct<br>
%               with three fields, weights, start_weights, stop_weights<br>
%         for a graph with |V| nodes and |E| arcs,<br>
%         DAG.weights: sparse double |V| x |V| matrix. Adjacency<br>
%               matrix. The non-zero entries represent costs on arcs<br>
%               linking two nodes.<br>
%         DAG.start_weights: dense double |V| vector. Represent the costs<br>
%               of starting a path from a specific node.<br>
%         DAG.stop_weights: dense double |V| vector. Represent the costs<br>
%               of ending a path at a specific node.<br>
%<br>
%         if param.regul='graph-path-l0', non-convex penalty<br>
%         if param.regul='graph-path-conv', convex penalty<br>
%<br>
%         param: struct<br>
%               param.lambda  (regularization parameter)<br>
%               param.regul (choice of regularization, see above)<br>
%               param.verbose (optional, verbosity level, false by default)<br>
%               param.intercept (optional, last row of U is not regularized,<br>
%                 false by default)<br>
%               param.pos (optional, adds positivity constraints on the<br>
%                 coefficients, false by default)<br>
%               param.precision (optional, by default a very large integer.<br>
%                 It returns approximate proximal operator by choosing a small integer,<br>
%                 for example, 100 or 1000.<br>
%               param.numThreads (optional, number of threads for exploiting<br>
%                 multi-core / multi-cpus. By default, it takes the value -1,<br>
%                 which automatically selects all the available CPUs/cores).<br>
%<br>
% Output: V: double p x n matrix (output coefficients)<br>
%         val_regularizer: double 1 x n vector (value of the regularization<br>
%         term at the optimum).<br>
%<br>
% Author: Julien Mairal, 2012</span></td></tr>
</table><p>
The following piece of code illustrates how to use this function.
</p><table class="lstframe c011"><tr><td class="mouselstlisting"><span class="c002">clear</span> <span class="c002">all</span>;<br>
<span class="c002">rand</span>(<span class="c003">'seed'</span>,0);<br>
<span class="c002">randn</span>(<span class="c003">'seed'</span>,0);<br>
<span class="c002">fprintf</span>(<span class="c003">'test mexProximalPathCoding\n'</span>);<br>
p=100;<br>
<span class="c001">% generate a DAG</span><br>
G=sprand(p,p,0.02);<br>
G=mexRemoveCyclesGraph(G);<br>
<span class="c002">fprintf</span>(<span class="c003">'\n'</span>);<br>
<br>
<span class="c001">% generate a data matrix</span><br>
U=<span class="c002">randn</span>(p,10);<br>
U=U-<span class="c002">mean</span>(U(:));<br>
U=mexNormalize(U);<br>
<br>
<span class="c001">% input graph</span><br>
graph.weights=G;<br>
graph.stop_weights=<span class="c002">zeros</span>(1,p);<br>
graph.start_weights=10*ones(1,p);<br>
<br>
<span class="c001">% FISTA parameters</span><br>
param.num_threads=-1; <span class="c001">% all cores (-1 by default)</span><br>
param.verbose=true;   <span class="c001">% verbosity, false by default</span><br>
param.lambda=0.05; <span class="c001">% regularization parameter</span><br>
<br>
<span class="c002">fprintf</span>(<span class="c003">'Proximal convex path penalty\n'</span>);<br>
param.regul=<span class="c003">'graph-path-conv'</span>;<br>
<span class="c002">tic</span><br>
[V1 optim]=mexProximalPathCoding(U,graph,param);<br>
t=<span class="c002">toc</span>;<br>
num=mexCountConnexComponents(graph.weights,V1(:,1));<br>
<span class="c002">fprintf</span>(<span class="c003">'Num of connected components: %d\n'</span>,num);<br>
<br>
<span class="c002">fprintf</span>(<span class="c003">'Proximal non-convex path penalty\n'</span>);<br>
param.regul=<span class="c003">'graph-path-l0'</span>;<br>
param.lambda=0.005;<br>
<span class="c002">tic</span><br>
[V2 optim]=mexProximalPathCoding(U,graph,param);<br>
t=<span class="c002">toc</span>;<br>
num=mexCountConnexComponents(graph.weights,V2(:,1));<br>
<span class="c002">fprintf</span>(<span class="c003">'Num of connected components: %d\n'</span>,num);<br>
<br>
graph.start_weights=1*ones(1,p);<br>
param.lambda=0.05;<br>
<span class="c002">fprintf</span>(<span class="c003">'Proximal convex path penalty\n'</span>);<br>
param.regul=<span class="c003">'graph-path-conv'</span>;<br>
<span class="c002">tic</span><br>
[V1 optim]=mexProximalPathCoding(U,graph,param);<br>
t=<span class="c002">toc</span>;<br>
num=mexCountConnexComponents(graph.weights,V1(:,1));<br>
<span class="c002">fprintf</span>(<span class="c003">'Num of connected components: %d\n'</span>,num);<br>
<br>
<span class="c002">fprintf</span>(<span class="c003">'Proximal non-convex path penalty\n'</span>);<br>
param.regul=<span class="c003">'graph-path-l0'</span>;<br>
param.lambda=0.005;<br>
<span class="c002">tic</span><br>
[V2 optim]=mexProximalPathCoding(U,graph,param);<br>
t=<span class="c002">toc</span>;<br>
num=mexCountConnexComponents(graph.weights,V2(:,1));<br>
<span class="c002">fprintf</span>(<span class="c003">'Num of connected components: %d\n'</span>,num);</td></tr>
</table><p>
This function is associated to a function to evaluate the penalties:
</p>
<!--TOC subsection id="sec29" Function mexEvalPathCoding-->
<h3 class="subsection" id="sec29">5.6  Function mexEvalPathCoding</h3><!--SEC END --><table class="lstframe c011"><tr><td class="mouselstlisting"><span class="c001">% <br>
% Usage:   [val [paths]]=mexEvalPathCoding(U,DAG,param);<br>
%<br>
% Name: mexEvalPathCoding<br>
%<br>
% Description: mexEvalPathCoding evaluate the path coding penalies <br>
%         of http://arxiv.org/abs/1204.4539 and provides a path <br>
%         decomposition of a vector W.<br>
%<br>
%         Given an input matrix U=[u^1,\ldots,u^n], <br>
%<br>
%<br>
% Inputs: U:  double p x n matrix   (input signals)<br>
%               m is the signal size<br>
%         DAG:  struct<br>
%               with three fields, weights, start_weights, stop_weights<br>
%         for a graph with |V| nodes and |E| arcs,<br>
%         DAG.weights: sparse double |V| x |V| matrix. Adjacency<br>
%               matrix. The non-zero entries represent costs on arcs<br>
%               linking two nodes.<br>
%         DAG.start_weights: dense double |V| vector. Represent the costs<br>
%               of starting a path from a specific node.<br>
%         DAG.stop_weights: dense double |V| vector. Represent the costs<br>
%               of ending a path at a specific node.<br>
%<br>
%         if param.regul='graph-path-l0', non-convex penalty<br>
%         if param.regul='graph-path-conv', convex penalty<br>
%<br>
%         param: struct<br>
%               param.regul (choice of regularization, see above)<br>
%               param.verbose (optional, verbosity level, false by default)<br>
%               param.precision (optional, by default a very large integer.<br>
%                 It returns approximate proximal operator by choosing a small integer,<br>
%                 for example, 100 or 1000.<br>
%               param.numThreads (optional, number of threads for exploiting<br>
%                 multi-core / multi-cpus. By default, it takes the value -1,<br>
%                 which automatically selects all the available CPUs/cores).<br>
%<br>
% Output: V: double 1 x n vector (values of the objective function)<br>
%         paths: optional, double sparse p x k matrix. selected paths for the <br>
%                first column of U<br>
%<br>
% Author: Julien Mairal, 2012</span></td></tr>
</table><p>
The following piece of code illustrates how to use this function.
</p><table class="lstframe c011"><tr><td class="mouselstlisting"><span class="c002">clear</span> <span class="c002">all</span>;<br>
<span class="c002">rand</span>(<span class="c003">'seed'</span>,0);<br>
<span class="c002">randn</span>(<span class="c003">'seed'</span>,0);<br>
<br>
p=100;<br>
G=sprand(p,p,0.05);<br>
G=mexRemoveCyclesGraph(G);<br>
<span class="c002">fprintf</span>(<span class="c003">'\n'</span>);<br>
<br>
<span class="c001">% input graph</span><br>
graph.weights=G;<br>
graph.stop_weights=<span class="c002">zeros</span>(1,p);<br>
graph.start_weights=10*ones(1,p);<br>
<br>
param.regul=<span class="c003">'graph-path-l0'</span>;<br>
U=<span class="c002">randn</span>(p,10);<br>
U=U-<span class="c002">mean</span>(U(:));<br>
U=mexNormalize(U);<br>
param.lambda=0.005;<br>
[V2 optim]=mexProximalPathCoding(U,graph,param);<br>
[vals paths]=mexEvalPathCoding(U,graph,param);</td></tr>
</table><p>After having presented the regularization terms which our software can handle,
we present the various formulations that we address
</p>
<!--TOC subsection id="sec30" Problems Addressed-->
<h3 class="subsection" id="sec30">5.7  Problems Addressed</h3><!--SEC END --><p>
We present here regression or classification formulations and their multi-task variants.
</p>
<!--TOC subsubsection id="sec31" Regression Problems with the Square Loss-->
<h4 class="subsubsection" id="sec31">5.7.1  Regression Problems with the Square Loss</h4><!--SEC END --><p> Given a training set {<span class="c009">x</span><sup><span class="c007">i</span></sup>,<span class="c007">y</span><sub><span class="c007">i</span></sub>}<sub><span class="c007">i</span>=1</sub><sup><span class="c007">n</span></sup>, with <span class="c009">x</span><sup><span class="c007">i</span></sup> ∈ ℝ<sup><span class="c007">p</span></sup> and <span class="c007">y</span><sub><span class="c007">i</span></sub> ∈ ℝ for all <span class="c007">i</span> in [ 1;<span class="c007">n</span> ], we address
</p><table class="display dcenter"><tr class="c016"><td class="dcell"><table class="display"><tr><td class="dcell c012">&nbsp;</td></tr>
<tr><td class="dcell c012">min</td></tr>
<tr><td class="dcell c012"><span class="c009">w</span> ∈ ℝ<sup><span class="c007">p</span></sup>, <span class="c007">b</span> ∈ ℝ</td></tr>
</table></td><td class="dcell"> </td><td class="dcell"><table class="display"><tr><td class="dcell c012"><span class="c007">n</span></td></tr>
<tr><td class="dcell c012"><span class="c006">∑</span></td></tr>
<tr><td class="dcell c012"><span class="c007">i</span>=1</td></tr>
</table></td><td class="dcell"> </td><td class="dcell"><table class="display"><tr><td class="dcell c012">1</td></tr>
<tr><td class="hbar"></td></tr>
<tr><td class="dcell c012">2</td></tr>
</table></td><td class="dcell">(<span class="c007">y</span><sub><span class="c007">i</span></sub>−<span class="c009">w</span><sup>⊤</sup><span class="c009">x</span><sup><span class="c007">i</span></sup>−<span class="c007">b</span>)<sup>2</sup> + λψ(<span class="c009">w</span>),
</td></tr>
</table><p>
where <span class="c007">b</span> is an optional variable acting as an “intercept”, which is not regularized, and ψ
can be any of the regularization functions presented above. 
Let us consider the vector <span class="c009">y</span> in ℝ<sup><span class="c007">n</span></sup> that carries the entries <span class="c007">y</span><sub><span class="c007">i</span></sub>. 
The problem without the intercept takes the following form, which we have already
encountered in the previous toolbox, but with different notations:
</p><table class="display dcenter"><tr class="c016"><td class="dcell"><table class="display"><tr><td class="dcell c012">&nbsp;</td></tr>
<tr><td class="dcell c012">min</td></tr>
<tr><td class="dcell c012"><span class="c009">w</span> ∈ ℝ<sup><span class="c007">p</span></sup></td></tr>
</table></td><td class="dcell"> </td><td class="dcell"><table class="display"><tr><td class="dcell c012">1</td></tr>
<tr><td class="hbar"></td></tr>
<tr><td class="dcell c012">2</td></tr>
</table></td><td class="dcell">||<span class="c009">y</span>−<span class="c009">Xw</span>||<sub>2</sub><sup>2</sup> + λψ(<span class="c009">w</span>),
</td></tr>
</table><p>
where the <span class="c009">X</span>=[<span class="c009">x</span><sup><span class="c007">i</span></sup>,…,<span class="c009">x</span><sup><span class="c007">n</span></sup>]<sup><span class="c007">T</span></sup> (the <span class="c009">x</span><sup><span class="c007">i</span></sup>’s are here the rows of <span class="c009">X</span>).
</p>
<!--TOC subsubsection id="sec32" Classification Problems with the Logistic Loss-->
<h4 class="subsubsection" id="sec32">5.7.2  Classification Problems with the Logistic Loss</h4><!--SEC END --><p>
The next formulation that our software can solve is the regularized logistic regression formulation.
We are again given a training set {<span class="c009">x</span><sup><span class="c007">i</span></sup>,<span class="c007">y</span><sub><span class="c007">i</span></sub>}<sub><span class="c007">i</span>=1</sub><sup><span class="c007">n</span></sup>, with <span class="c009">x</span><sup><span class="c007">i</span></sup> ∈
ℝ<sup><span class="c007">p</span></sup>, but the variables <span class="c007">y</span><sub><span class="c007">i</span></sub> are now in {−1,+1} for all <span class="c007">i</span> in
[ 1;<span class="c007">n</span> ]. The optimization problem we address is
</p><table class="display dcenter"><tr class="c016"><td class="dcell">   </td><td class="dcell"><table class="display"><tr><td class="dcell c012">&nbsp;</td></tr>
<tr><td class="dcell c012">min</td></tr>
<tr><td class="dcell c012"><span class="c009">w</span> ∈ ℝ<sup><span class="c007">p</span></sup>, <span class="c007">b</span> ∈ ℝ</td></tr>
</table></td><td class="dcell"> </td><td class="dcell"><table class="display"><tr><td class="dcell c012">1</td></tr>
<tr><td class="hbar"></td></tr>
<tr><td class="dcell c012"><span class="c007">n</span></td></tr>
</table></td><td class="dcell"><table class="display"><tr><td class="dcell c012"><span class="c007">n</span></td></tr>
<tr><td class="dcell c012"><span class="c006">∑</span></td></tr>
<tr><td class="dcell c012"><span class="c007">i</span>=1</td></tr>
</table></td><td class="dcell"> log(1+<span class="c007">e</span><sup>−<span class="c007">y</span><sub><span class="c007">i</span></sub>(<span class="c009">w</span><sup>⊤</sup><span class="c009">x</span><sup><span class="c007">i</span></sup>+<span class="c007">b</span>)</sup> + λψ(<span class="c009">w</span>),
</td></tr>
</table><p>
with again ψ taken to be one of the regularization function presented above, and <span class="c007">b</span> is an optional intercept.
</p>
<!--TOC subsubsection id="sec33" Multi-class Classification Problems with the Softmax Loss-->
<h4 class="subsubsection" id="sec33">5.7.3  Multi-class Classification Problems with the Softmax Loss</h4><!--SEC END --><p>
We have also implemented a multi-class logistic classifier (or softmax).
For a classification problem with <span class="c007">r</span> classes, we are given a training set {<span class="c009">x</span><sup><span class="c007">i</span></sup>,<span class="c007">y</span><sub><span class="c007">i</span></sub>}<sub><span class="c007">i</span>=1</sub><sup><span class="c007">n</span></sup>, where the variables <span class="c009">x</span><sup><span class="c007">i</span></sup> are still vectors in ℝ<sup><span class="c007">p</span></sup>, but the <span class="c007">y</span><sub><span class="c007">i</span></sub>’s have integer values in {1,2,…,<span class="c007">r</span>}. The formulation we address is the following multi-class learning problem
</p><table class="display dcenter"><tr class="c016"><td class="dcell">
</td><td class="dcell"><table class="display"><tr><td class="dcell c012">&nbsp;</td></tr>
<tr><td class="dcell c012">min</td></tr>
<tr><td class="dcell c012"><span class="c009">W</span> ∈ ℝ<sup><span class="c007">p</span> × <span class="c007">r</span></sup>, <span class="c009">b</span> ∈ ℝ<sup><span class="c007">r</span></sup></td></tr>
</table></td><td class="dcell"> </td><td class="dcell"><table class="display"><tr><td class="dcell c012">1</td></tr>
<tr><td class="hbar"></td></tr>
<tr><td class="dcell c012"><span class="c007">n</span></td></tr>
</table></td><td class="dcell"><table class="display"><tr><td class="dcell c012"><span class="c007">n</span></td></tr>
<tr><td class="dcell c012"><span class="c006">∑</span></td></tr>
<tr><td class="dcell c012"><span class="c007">i</span>=1</td></tr>
</table></td><td class="dcell"> log</td><td class="dcell">⎛<br>
⎜<br>
⎝</td><td class="dcell"><table class="display"><tr><td class="dcell c012"><span class="c007">r</span></td></tr>
<tr><td class="dcell c012"><span class="c006">∑</span></td></tr>
<tr><td class="dcell c012"><span class="c007">j</span>=1</td></tr>
</table></td><td class="dcell"> <span class="c007">e</span><sup> (<span class="c009">w</span><sup><span class="c007">j</span></sup>−<span class="c009">w</span><sup><span class="c009">y</span><sub><span class="c007">i</span></sub></sup>)<sup>⊤</sup><span class="c009">x</span><sup><span class="c007">i</span></sup> + <span class="c009">b</span><sub><span class="c007">j</span></sub>−<span class="c009">b</span><sub><span class="c009">y</span><sub><span class="c007">i</span></sub></sub></sup></td><td class="dcell">⎞<br>
⎟<br>
⎠</td><td class="dcell">+ λ</td><td class="dcell"><table class="display"><tr><td class="dcell c012"><span class="c007">r</span></td></tr>
<tr><td class="dcell c012"><span class="c006">∑</span></td></tr>
<tr><td class="dcell c012"><span class="c007">j</span>=1</td></tr>
</table></td><td class="dcell">ψ(<span class="c009">w</span><sup><span class="c007">j</span></sup>),<a id="software:eq:class"></a>
    (46)</td></tr>
</table><p>
where <span class="c009">W</span> = [<span class="c009">w</span><sup>1</sup>,…,<span class="c009">w</span><sup><span class="c007">r</span></sup>] and the optional vector <span class="c009">b</span> in ℝ<sup><span class="c007">r</span></sup> carries intercepts for each class.
</p>
<!--TOC subsubsection id="sec34" Multi-task Regression Problems with the Square Loss-->
<h4 class="subsubsection" id="sec34">5.7.4  Multi-task Regression Problems with the Square Loss</h4><!--SEC END --><p>
We are now considering a problem with <span class="c007">r</span> tasks, and a training set
{<span class="c009">x</span><sup><span class="c007">i</span></sup>,<span class="c009">y</span><sup><span class="c007">i</span></sup>}<sub><span class="c007">i</span>=1</sub><sup><span class="c007">n</span></sup>, where the variables <span class="c009">x</span><sup><span class="c007">i</span></sup> are still vectors in ℝ<sup><span class="c007">p</span></sup>, and <span class="c009">y</span><sup><span class="c007">i</span></sup>
is a vector in ℝ<sup><span class="c007">r</span></sup>. We are looking for <span class="c007">r</span> regression vectors <span class="c009">w</span><sup><span class="c007">j</span></sup>, for <span class="c007">j</span>∈ [ 1;<span class="c007">r</span> ], or equivalently for a matrix <span class="c009">W</span>=[<span class="c009">w</span><sup>1</sup>,…,<span class="c009">w</span><sup><span class="c007">r</span></sup>] in ℝ<sup><span class="c007">p</span> × <span class="c007">r</span></sup>. The formulation we address is the following
multi-task regression problem
</p><table class="display dcenter"><tr class="c016"><td class="dcell">         </td><td class="dcell"><table class="display"><tr><td class="dcell c012">&nbsp;</td></tr>
<tr><td class="dcell c012">min</td></tr>
<tr><td class="dcell c012"><span class="c009">W</span> ∈ ℝ<sup><span class="c007">p</span> × <span class="c007">r</span></sup>, <span class="c009">b</span> ∈ ℝ<sup><span class="c007">r</span></sup></td></tr>
</table></td><td class="dcell"> </td><td class="dcell"><table class="display"><tr><td class="dcell c012"><span class="c007">r</span></td></tr>
<tr><td class="dcell c012"><span class="c006">∑</span></td></tr>
<tr><td class="dcell c012"><span class="c007">j</span>=1</td></tr>
</table></td><td class="dcell"><table class="display"><tr><td class="dcell c012"><span class="c007">n</span></td></tr>
<tr><td class="dcell c012"><span class="c006">∑</span></td></tr>
<tr><td class="dcell c012"><span class="c007">i</span>=1</td></tr>
</table></td><td class="dcell"> </td><td class="dcell"><table class="display"><tr><td class="dcell c012">1</td></tr>
<tr><td class="hbar"></td></tr>
<tr><td class="dcell c012">2</td></tr>
</table></td><td class="dcell">(<span class="c009">y</span><sub><span class="c007">j</span></sub><sup><span class="c007">i</span></sup>−<span class="c009">w</span><sup>⊤</sup><span class="c009">x</span><sup><span class="c007">i</span></sup>−<span class="c009">b</span><sub><span class="c007">j</span></sub>)<sup>2</sup> + λψ(<span class="c009">W</span>),
</td></tr>
</table><p>
where ψ is any of the regularization function on matrices we have presented in the previous section.
Note that by introducing the appropriate variables <span class="c009">Y</span>, the problem without intercept could be equivalently rewritten
</p><table class="display dcenter"><tr class="c016"><td class="dcell">   </td><td class="dcell"><table class="display"><tr><td class="dcell c012">&nbsp;</td></tr>
<tr><td class="dcell c012">min</td></tr>
<tr><td class="dcell c012"><span class="c009">W</span> ∈ ℝ<sup><span class="c007">p</span> × <span class="c007">r</span></sup></td></tr>
</table></td><td class="dcell"> </td><td class="dcell"><table class="display"><tr><td class="dcell c012">1</td></tr>
<tr><td class="hbar"></td></tr>
<tr><td class="dcell c012">2</td></tr>
</table></td><td class="dcell"> ||<span class="c009">Y</span>−<span class="c009">XW</span>||<sub>F</sub><sup>2</sup> + λψ(<span class="c009">W</span>).
</td></tr>
</table>
<!--TOC subsubsection id="sec35" Multi-task Classification Problems with the Logistic Loss-->
<h4 class="subsubsection" id="sec35">5.7.5  Multi-task Classification Problems with the Logistic Loss</h4><!--SEC END --><p>
The multi-task version of the logistic regression follows the same principle.
We consider <span class="c007">r</span> tasks, and a training set
{<span class="c009">x</span><sup><span class="c007">i</span></sup>,<span class="c009">y</span><sup><span class="c007">i</span></sup>}<sub><span class="c007">i</span>=1</sub><sup><span class="c007">n</span></sup>, with the <span class="c009">x</span><sup><span class="c007">i</span></sup>’s in ℝ<sup><span class="c007">p</span></sup>, and the <span class="c009">y</span><sup><span class="c007">i</span></sup>’s
are vectors in {−1,+1}<sup><span class="c007">r</span></sup>. We look for a matrix <span class="c009">W</span>=[<span class="c009">w</span><sup>1</sup>,…,<span class="c009">w</span><sup><span class="c007">r</span></sup>] in ℝ<sup><span class="c007">p</span> × <span class="c007">r</span></sup>. The formulation is the following
multi-task regression problem
</p><table class="display dcenter"><tr class="c016"><td class="dcell">   </td><td class="dcell"><table class="display"><tr><td class="dcell c012">&nbsp;</td></tr>
<tr><td class="dcell c012">min</td></tr>
<tr><td class="dcell c012"><span class="c009">W</span> ∈ ℝ<sup><span class="c007">p</span> × <span class="c007">r</span></sup>, <span class="c009">b</span> ∈ ℝ<sup><span class="c007">r</span></sup></td></tr>
</table></td><td class="dcell"> </td><td class="dcell"><table class="display"><tr><td class="dcell c012"><span class="c007">r</span></td></tr>
<tr><td class="dcell c012"><span class="c006">∑</span></td></tr>
<tr><td class="dcell c012"><span class="c007">j</span>=1</td></tr>
</table></td><td class="dcell"><table class="display"><tr><td class="dcell c012">1</td></tr>
<tr><td class="hbar"></td></tr>
<tr><td class="dcell c012"><span class="c007">n</span></td></tr>
</table></td><td class="dcell"><table class="display"><tr><td class="dcell c012"><span class="c007">n</span></td></tr>
<tr><td class="dcell c012"><span class="c006">∑</span></td></tr>
<tr><td class="dcell c012"><span class="c007">i</span>=1</td></tr>
</table></td><td class="dcell"> log</td><td class="dcell">⎛<br>
⎜<br>
⎝</td><td class="dcell">1+<span class="c007">e</span><sup>−<span class="c009">y</span><sub><span class="c007">j</span></sub><sup><span class="c007">i</span></sup>(<span class="c009">w</span><sup>⊤</sup><span class="c009">x</span><sup><span class="c007">i</span></sup>+<span class="c009">b</span><sub><span class="c007">j</span></sub>)</sup></td><td class="dcell">⎞<br>
⎟<br>
⎠</td><td class="dcell">+ λψ(<span class="c009">W</span>).
</td></tr>
</table>
<!--TOC subsubsection id="sec36" Multi-task and Multi-class Classification Problems with the Softmax Loss-->
<h4 class="subsubsection" id="sec36">5.7.6  Multi-task and Multi-class Classification Problems with the Softmax Loss</h4><!--SEC END --><p>
The multi-task/multi-class version directly follows from the formulation of Eq. (<a href="#software%3Aeq%3Aclass">46</a>), but associates with each class a task, and as a consequence, regularizes the matrix <span class="c009">W</span> in a particular way:
</p><table class="display dcenter"><tr class="c016"><td class="dcell">   </td><td class="dcell"><table class="display"><tr><td class="dcell c012">&nbsp;</td></tr>
<tr><td class="dcell c012">min</td></tr>
<tr><td class="dcell c012"><span class="c009">W</span> ∈ ℝ<sup><span class="c007">p</span> × <span class="c007">r</span></sup>, <span class="c009">b</span> ∈ ℝ<sup><span class="c007">r</span></sup></td></tr>
</table></td><td class="dcell"> </td><td class="dcell"><table class="display"><tr><td class="dcell c012">1</td></tr>
<tr><td class="hbar"></td></tr>
<tr><td class="dcell c012"><span class="c007">n</span></td></tr>
</table></td><td class="dcell"><table class="display"><tr><td class="dcell c012"><span class="c007">n</span></td></tr>
<tr><td class="dcell c012"><span class="c006">∑</span></td></tr>
<tr><td class="dcell c012"><span class="c007">i</span>=1</td></tr>
</table></td><td class="dcell"> log</td><td class="dcell">⎛<br>
⎜<br>
⎝</td><td class="dcell"><table class="display"><tr><td class="dcell c012"><span class="c007">r</span></td></tr>
<tr><td class="dcell c012"><span class="c006">∑</span></td></tr>
<tr><td class="dcell c012"><span class="c007">j</span>=1</td></tr>
</table></td><td class="dcell"> <span class="c007">e</span><sup> (<span class="c009">w</span><sup><span class="c007">j</span></sup>−<span class="c009">w</span><sup><span class="c009">y</span><sub><span class="c007">i</span></sub></sup>)<sup>⊤</sup><span class="c009">x</span><sup><span class="c007">i</span></sup> + <span class="c009">b</span><sub><span class="c007">j</span></sub>−<span class="c009">b</span><sub><span class="c009">y</span><sub><span class="c007">i</span></sub></sub></sup></td><td class="dcell">⎞<br>
⎟<br>
⎠</td><td class="dcell">+ λψ(<span class="c009">W</span>).
</td></tr>
</table><p>
How duality gaps are computed for any of these formulations is presented in Appendix <a href="#appendix">A</a>.
We now present the main functions for solving these problems</p>
<!--TOC subsection id="sec37" Function mexFistaFlat-->
<h3 class="subsection" id="sec37">5.8  Function mexFistaFlat</h3><!--SEC END --><p>
Given a matrix <span class="c009">X</span>=[<span class="c009">x</span><sup>1</sup>,…,<span class="c009">x</span><sup><span class="c007">p</span></sup>]<sup><span class="c007">T</span></sup> in ℝ<sup><span class="c007">m</span> × <span class="c007">p</span></sup>, and a matrix <span class="c009">Y</span>=[<span class="c009">y</span><sup>1</sup>,…,<span class="c009">y</span><sup><span class="c007">n</span></sup>], it solves the optimization problems presented in the previous section, with the same regularization functions as mexProximalFlat.
see usage details below:
</p><table class="lstframe c011"><tr><td class="mouselstlisting"><span class="c001">% <br>
% Usage: [W [optim]]=mexFistaFlat(Y,X,W0,param);<br>
%<br>
% Name: mexFistaFlat<br>
%<br>
% Description: mexFistaFlat solves sparse regularized problems.<br>
%         X is a design matrix of size m x p<br>
%         X=[x^1,...,x^n]', where the x_i's are the rows of X<br>
%         Y=[y^1,...,y^n] is a matrix of size m x n<br>
%         It implements the algorithms FISTA, ISTA and subgradient descent.<br>
%         <br>
%           - if param.loss='square' and param.regul is a regularization function for vectors,<br>
%             the entries of Y are real-valued,  W = [w^1,...,w^n] is a matrix of size p x n<br>
%             For all column y of Y, it computes a column w of W such that<br>
%               w = argmin 0.5||y- X w||_2^2 + lambda psi(w)<br>
%<br>
%           - if param.loss='square' and param.regul is a regularization function for matrices<br>
%             the entries of Y are real-valued,  W is a matrix of size p x n. <br>
%             It computes the matrix W such that<br>
%               W = argmin 0.5||Y- X W||_F^2 + lambda psi(W)<br>
%            <br>
%           - param.loss='square-missing' : same as param.loss='square', but handles missing data<br>
%             represented by NaN (not a number) in the matrix Y<br>
%<br>
%           - if param.loss='logistic' and param.regul is a regularization function for vectors,<br>
%             the entries of Y are either -1 or +1, W = [w^1,...,w^n] is a matrix of size p x n<br>
%             For all column y of Y, it computes a column w of W such that<br>
%               w = argmin (1/m)sum_{j=1}^m log(1+e^(-y_j x^j' w)) + lambda psi(w),<br>
%             where x^j is the j-th row of X.<br>
%<br>
%           - if param.loss='logistic' and param.regul is a regularization function for matrices<br>
%             the entries of Y are either -1 or +1, W is a matrix of size p x n<br>
%               W = argmin sum_{i=1}^n(1/m)sum_{j=1}^m log(1+e^(-y^i_j x^j' w^i)) + lambda psi(W)<br>
%<br>
%           - if param.loss='multi-logistic' and param.regul is a regularization function for vectors,<br>
%             the entries of Y are in {0,1,...,N} where N is the total number of classes<br>
%             W = [W^1,...,W^n] is a matrix of size p x Nn, each submatrix W^i is of size p x N<br>
%             for all submatrix WW of W, and column y of Y, it computes<br>
%               WW = argmin (1/m)sum_{j=1}^m log(sum_{j=1}^r e^(x^j'(ww^j-ww^{y_j}))) + lambda sum_{j=1}^N psi(ww^j),<br>
%             where ww^j is the j-th column of WW.<br>
%<br>
%           - if param.loss='multi-logistic' and param.regul is a regularization function for matrices,<br>
%             the entries of Y are in {0,1,...,N} where N is the total number of classes<br>
%             W is a matrix of size p x N, it computes<br>
%               W = argmin (1/m)sum_{j=1}^m log(sum_{j=1}^r e^(x^j'(w^j-w^{y_j}))) + lambda psi(W)<br>
%             where ww^j is the j-th column of WW.<br>
%<br>
%           - param.loss='cur' : useful to perform sparse CUR matrix decompositions, <br>
%               W = argmin 0.5||Y-X*W*X||_F^2 + lambda psi(W)<br>
%<br>
%<br>
%         The function psi are those used by mexProximalFlat (see documentation)<br>
%<br>
%         This function can also handle intercepts (last row of W is not regularized),<br>
%         and/or non-negativity constraints on W, and sparse matrices for X<br>
%<br>
% Inputs: Y:  double dense m x n matrix<br>
%         X:  double dense or sparse m x p matrix   <br>
%         W0:  double dense p x n matrix or p x Nn matrix (for multi-logistic loss)<br>
%              initial guess<br>
%         param: struct<br>
%            param.loss (choice of loss, see above)<br>
%            param.regul (choice of regularization, see function mexProximalFlat)<br>
%            param.lambda (regularization parameter)<br>
%            param.lambda2 (optional, regularization parameter, 0 by default)<br>
%            param.lambda3 (optional, regularization parameter, 0 by default)<br>
%            param.verbose (optional, verbosity level, false by default)<br>
%            param.pos (optional, adds positivity constraints on the<br>
%                coefficients, false by default)<br>
%            param.transpose (optional, transpose the matrix in the regularization function)<br>
%            param.size_group (optional, for regularization functions assuming a group<br>
%                 structure)<br>
%            param.groups (int32, optional, for regularization functions assuming a group<br>
%                 structure, see mexProximalFlat)<br>
%            param.numThreads (optional, number of threads for exploiting<br>
%                multi-core / multi-cpus. By default, it takes the value -1,<br>
%                which automatically selects all the available CPUs/cores).<br>
%            param.max_it (optional, maximum number of iterations, 100 by default)<br>
%            param.it0 (optional, frequency for computing duality gap, every 10 iterations by default)<br>
%            param.tol (optional, tolerance for stopping criteration, which is a relative duality gap<br>
%                if it is available, or a relative change of parameters).<br>
%            param.gamma (optional, multiplier for increasing the parameter L in fista, 1.5 by default)<br>
%            param.L0 (optional, initial parameter L in fista, 0.1 by default, should be small enough)<br>
%            param.fixed_step (deactive the line search for L in fista and use param.L0 instead)<br>
%            param.linesearch_mode (line-search scheme when ista=true:<br>
%                      0: default, monotonic backtracking scheme<br>
%                      1: monotonic backtracking scheme, with restart at each iteration<br>
%                      2: Barzilai-Borwein step sizes (similar to SparSA by Wright et al.)<br>
%                      3: non-monotonic backtracking<br>
%            param.compute_gram (optional, pre-compute X^TX, false by default).<br>
%            param.intercept (optional, do not regularize last row of W, false by default).<br>
%            param.ista (optional, use ista instead of fista, false by default).<br>
%            param.subgrad (optional, if not param.ista, use subradient descent instead of fista, false by default).<br>
%            param.a, param.b (optional, if param.subgrad, the gradient step is a/(t+b)<br>
%            also similar options as mexProximalFlat<br>
%<br>
%            the function also implements the ADMM algorithm via an option param.admm=true. It is not documented<br>
%            and you need to look at the source code to use it.<br>
%<br>
% Output:  W:  double dense p x n matrix or p x Nn matrix (for multi-logistic loss)<br>
%          optim: optional, double dense 4 x n matrix.<br>
%              first row: values of the objective functions.<br>
%              third row: values of the relative duality gap (if available)<br>
%              fourth row: number of iterations<br>
%<br>
% Author: Julien Mairal, 2010</span></td></tr>
</table><p>
The following piece of code illustrates how to use this function.
</p><table class="lstframe c011"><tr><td class="mouselstlisting"><span class="c002">format</span> compact;<br>
<span class="c002">randn</span>(<span class="c003">'seed'</span>,0);<br>
param.numThreads=-1; <span class="c001">% all cores (-1 by default)</span><br>
param.verbose=true;   <span class="c001">% verbosity, false by default</span><br>
param.lambda=0.05; <span class="c001">% regularization parameter</span><br>
param.it0=10;      <span class="c001">% frequency for duality gap computations</span><br>
param.max_it=200; <span class="c001">% maximum number of iterations</span><br>
param.L0=0.1;<br>
param.tol=1e-3;<br>
param.intercept=false;<br>
param.pos=false;<br>
param.ista=false;<br>
<br>
X=<span class="c002">randn</span>(100,200);<br>
X=X-repmat(<span class="c002">mean</span>(X),[<span class="c002">size</span>(X,1) 1]);<br>
X=mexNormalize(X);<br>
Y=<span class="c002">randn</span>(100,1);<br>
Y=Y-repmat(<span class="c002">mean</span>(Y),[<span class="c002">size</span>(Y,1) 1]);<br>
Y=mexNormalize(Y);<br>
W0=<span class="c002">zeros</span>(<span class="c002">size</span>(X,2),<span class="c002">size</span>(Y,2));<br>
<span class="c001">% Regression experiments <br>
% 100 regression problems with the same design matrix X.</span><br>
<span class="c002">fprintf</span>(<span class="c003">'\nVarious regression experiments\n'</span>);<br>
param.compute_gram=true;<br>
<span class="c002">fprintf</span>(<span class="c003">'\nFISTA + Regression l1\n'</span>);<br>
param.loss=<span class="c003">'square'</span>;<br>
param.regul=<span class="c003">'l1'</span>;<br>
<span class="c001">% param.regul='group-lasso-l2';<br>
% param.size_group=10;</span><br>
<span class="c002">tic</span><br>
[W optim_info]=mexFistaFlat(Y,X,W0,param);<br>
t=<span class="c002">toc</span>;<br>
<span class="c002">fprintf</span>(<span class="c003">'mean loss: %f, mean relative duality_gap: %f, time: %f, number of iterations: %f\n'</span>,<span class="c002">mean</span>(optim_info(1,:)),<span class="c002">mean</span>(optim_info(3,:)),t,<span class="c002">mean</span>(optim_info(4,:)));<br>
<br>
param.regul=<span class="c003">'l1'</span>;<br>
<span class="c002">fprintf</span>(<span class="c003">'\nISTA + Regression l1\n'</span>);<br>
param.ista=true;<br>
<span class="c002">tic</span><br>
[W optim_info]=mexFistaFlat(Y,X,W0,param);<br>
t=<span class="c002">toc</span>;<br>
<span class="c002">fprintf</span>(<span class="c003">'mean loss: %f, mean relative duality_gap: %f, time: %f, number of iterations: %f\n'</span>,<span class="c002">mean</span>(optim_info(1,:)),<span class="c002">mean</span>(optim_info(3,:)),t,<span class="c002">mean</span>(optim_info(4,:)));<br>
<br>
param.regul=<span class="c003">'l1'</span>;<br>
<span class="c002">fprintf</span>(<span class="c003">'\nISTA + Regression l1 + Barzilai Borwein (SPARSA)\n'</span>);<br>
param.ista=true;<br>
param.barzilaiborwein=true;<br>
<span class="c002">tic</span><br>
[W optim_info]=mexFistaFlat(Y,X,W0,param);<br>
t=<span class="c002">toc</span>;<br>
<span class="c002">fprintf</span>(<span class="c003">'mean loss: %f, mean relative duality_gap: %f, time: %f, number of iterations: %f\n'</span>,<span class="c002">mean</span>(optim_info(1,:)),<span class="c002">mean</span>(optim_info(3,:)),t,<span class="c002">mean</span>(optim_info(4,:)));<br>
param.barzilaiborwein=false;<br>
<br>
<br>
<span class="c002">fprintf</span>(<span class="c003">'\nSubgradient Descent + Regression l1\n'</span>);<br>
param.ista=false;<br>
param.subgrad=true;<br>
param.a=0.1;<br>
param.b=1000; <span class="c001">% arbitrary parameters</span><br>
max_it=param.max_it;<br>
it0=param.it0;<br>
param.max_it=500;<br>
param.it0=50;<br>
<span class="c002">tic</span><br>
[W optim_info]=mexFistaFlat(Y,X,W0,param);<br>
t=<span class="c002">toc</span>;<br>
param.subgrad=false;<br>
param.max_it=max_it;<br>
param.it0=it0;<br>
<span class="c002">fprintf</span>(<span class="c003">'mean loss: %f, mean relative duality_gap: %f, time: %f, number of iterations: %f\n'</span>,<span class="c002">mean</span>(optim_info(1,:)),<span class="c002">mean</span>(optim_info(3,:)),t,<span class="c002">mean</span>(optim_info(4,:)));<br>
<br>
<span class="c002">fprintf</span>(<span class="c003">'\nFISTA + Regression l2\n'</span>);<br>
param.regul=<span class="c003">'l2'</span>;<br>
<span class="c002">tic</span><br>
[W optim_info]=mexFistaFlat(Y,X,W0,param);<br>
t=<span class="c002">toc</span>;<br>
<span class="c002">fprintf</span>(<span class="c003">'mean loss: %f, mean relative duality_gap: %f, time: %f, number of iterations: %f\n'</span>,<span class="c002">mean</span>(optim_info(1,:)),<span class="c002">mean</span>(optim_info(3,:)),t,<span class="c002">mean</span>(optim_info(4,:)));<br>
<br>
<span class="c002">fprintf</span>(<span class="c003">'\nFISTA + Regression l2 + sparse feature matrix\n'</span>);<br>
param.regul=<span class="c003">'l2'</span>;<br>
<span class="c002">tic</span><br>
[W optim_info]=mexFistaFlat(Y,<span class="c002">sparse</span>(X),W0,param);<br>
t=<span class="c002">toc</span>;<br>
<span class="c002">fprintf</span>(<span class="c003">'mean loss: %f, mean relative duality_gap: %f, time: %f, number of iterations: %f\n'</span>,<span class="c002">mean</span>(optim_info(1,:)),<span class="c002">mean</span>(optim_info(3,:)),t,<span class="c002">mean</span>(optim_info(4,:)));<br>
<br>
<br>
<span class="c002">fprintf</span>(<span class="c003">'\nFISTA + Regression Elastic-Net\n'</span>);<br>
param.regul=<span class="c003">'elastic-net'</span>;<br>
param.lambda2=0.1;<br>
<span class="c002">tic</span><br>
[W optim_info]=mexFistaFlat(Y,X,W0,param);<br>
t=<span class="c002">toc</span>;<br>
<span class="c002">fprintf</span>(<span class="c003">'mean loss: %f, time: %f, number of iterations: %f\n'</span>,<span class="c002">mean</span>(optim_info(1,:)),t,<span class="c002">mean</span>(optim_info(4,:)));<br>
<br>
<span class="c002">fprintf</span>(<span class="c003">'\nFISTA + Group Lasso L2\n'</span>);<br>
param.regul=<span class="c003">'group-lasso-l2'</span>;<br>
param.size_group=2;  <span class="c001">% all the groups are of size 2</span><br>
<span class="c002">tic</span><br>
[W optim_info]=mexFistaFlat(Y,X,W0,param);<br>
t=<span class="c002">toc</span>;<br>
<span class="c002">fprintf</span>(<span class="c003">'mean loss: %f, mean relative duality_gap: %f, time: %f, number of iterations: %f\n'</span>,<span class="c002">mean</span>(optim_info(1,:)),<span class="c002">mean</span>(optim_info(3,:)),t,<span class="c002">mean</span>(optim_info(4,:)));<br>
<br>
<span class="c002">fprintf</span>(<span class="c003">'\nFISTA + Group Lasso L2 with variable size of groups \n'</span>);<br>
param.regul=<span class="c003">'group-lasso-l2'</span>;<br>
param2=param;<br>
param2.groups=int32(randi(5,1,<span class="c002">size</span>(X,2)));  <span class="c001">% all the groups are of size 2</span><br>
param2.lambda=10*param2.lambda;<br>
<span class="c002">tic</span><br>
[W optim_info]=mexFistaFlat(Y,X,W0,param2);<br>
t=<span class="c002">toc</span>;<br>
<span class="c002">fprintf</span>(<span class="c003">'mean loss: %f, mean relative duality_gap: %f, time: %f, number of iterations: %f\n'</span>,<span class="c002">mean</span>(optim_info(1,:)),<span class="c002">mean</span>(optim_info(3,:)),t,<span class="c002">mean</span>(optim_info(4,:)));<br>
<br>
<span class="c002">fprintf</span>(<span class="c003">'\nFISTA + Trace Norm\n'</span>);<br>
param.regul=<span class="c003">'trace-norm-vec'</span>;<br>
param.size_group=5;<br>
<span class="c002">tic</span><br>
[W optim_info]=mexFistaFlat(Y,X,W0,param);<br>
t=<span class="c002">toc</span>;<br>
<span class="c002">fprintf</span>(<span class="c003">'mean loss: %f, mean relative duality_gap: %f, time: %f, number of iterations: %f\n'</span>,<span class="c002">mean</span>(optim_info(1,:)),<span class="c002">mean</span>(optim_info(3,:)),t,<span class="c002">mean</span>(optim_info(4,:)));<br>
<br>
<span class="c002">fprintf</span>(<span class="c003">'\nFISTA + Regression Fused-Lasso\n'</span>);<br>
param.regul=<span class="c003">'fused-lasso'</span>;<br>
param.lambda2=0.1;<br>
param.lambda3=0.1; <span class="c001">%</span><br>
<span class="c002">tic</span><br>
[W optim_info]=mexFistaFlat(Y,X,W0,param);<br>
t=<span class="c002">toc</span>;<br>
<span class="c002">fprintf</span>(<span class="c003">'mean loss: %f, time: %f, number of iterations: %f\n'</span>,<span class="c002">mean</span>(optim_info(1,:)),t,<span class="c002">mean</span>(optim_info(4,:)));<br>
<br>
<span class="c002">fprintf</span>(<span class="c003">'\nFISTA + Regression no regularization\n'</span>);<br>
param.regul=<span class="c003">'none'</span>;<br>
<span class="c002">tic</span><br>
[W optim_info]=mexFistaFlat(Y,X,W0,param);<br>
t=<span class="c002">toc</span>;<br>
<span class="c002">fprintf</span>(<span class="c003">'mean loss: %f, time: %f, number of iterations: %f\n'</span>,<span class="c002">mean</span>(optim_info(1,:)),t,<span class="c002">mean</span>(optim_info(4,:)));<br>
<br>
<span class="c002">fprintf</span>(<span class="c003">'\nFISTA + Regression l1 with intercept \n'</span>);<br>
param.intercept=true;<br>
param.regul=<span class="c003">'l1'</span>;<br>
<span class="c002">tic</span><br>
[W optim_info]=mexFistaFlat(Y,[X ones(<span class="c002">size</span>(X,1),1)],[W0; <span class="c002">zeros</span>(1,<span class="c002">size</span>(W0,2))],param); <span class="c001">% adds a column of ones to X for the intercept</span><br>
t=<span class="c002">toc</span>;<br>
<span class="c002">fprintf</span>(<span class="c003">'mean loss: %f, mean relative duality_gap: %f, time: %f, number of iterations: %f\n'</span>,<span class="c002">mean</span>(optim_info(1,:)),<span class="c002">mean</span>(optim_info(3,:)),t,<span class="c002">mean</span>(optim_info(4,:)));<br>
<br>
<span class="c002">fprintf</span>(<span class="c003">'\nFISTA + Regression l1 with intercept+ non-negative \n'</span>);<br>
param.pos=true;<br>
param.regul=<span class="c003">'l1'</span>;<br>
<span class="c002">tic</span><br>
[W optim_info]=mexFistaFlat(Y,[X ones(<span class="c002">size</span>(X,1),1)],[W0; <span class="c002">zeros</span>(1,<span class="c002">size</span>(W0,2))],param);<br>
t=<span class="c002">toc</span>;<br>
<span class="c002">fprintf</span>(<span class="c003">'mean loss: %f, time: %f, number of iterations: %f\n'</span>,<span class="c002">mean</span>(optim_info(1,:)),t,<span class="c002">mean</span>(optim_info(4,:)));<br>
param.pos=false;<br>
param.intercept=false;<br>
<br>
<span class="c002">fprintf</span>(<span class="c003">'\nISTA + Regression l0\n'</span>);<br>
param.regul=<span class="c003">'l0'</span>;<br>
<span class="c002">tic</span><br>
[W optim_info]=mexFistaFlat(Y,X,W0,param);<br>
t=<span class="c002">toc</span>;<br>
<span class="c002">fprintf</span>(<span class="c003">'mean loss: %f, time: %f, number of iterations: %f\n'</span>,<span class="c002">mean</span>(optim_info(1,:)),t,<span class="c002">mean</span>(optim_info(4,:)));<br>
<br>
<span class="c002">fprintf</span>(<span class="c003">'\nOne classification experiment\n'</span>);<br>
Y=2*double(<span class="c002">randn</span>(100,1) &gt; 0)-1;<br>
<span class="c002">fprintf</span>(<span class="c003">'\nFISTA + Logistic l1\n'</span>);<br>
param.regul=<span class="c003">'l1'</span>;<br>
param.loss=<span class="c003">'logistic'</span>;<br>
param.lambda=0.01;<br>
<span class="c002">tic</span><br>
[W optim_info]=mexFistaFlat(Y,X,W0,param);<br>
t=<span class="c002">toc</span>;<br>
<span class="c002">fprintf</span>(<span class="c003">'mean loss: %f, mean relative duality_gap: %f, time: %f, number of iterations: %f\n'</span>,<span class="c002">mean</span>(optim_info(1,:)),<span class="c002">mean</span>(optim_info(3,:)),t,<span class="c002">mean</span>(optim_info(4,:)));<br>
<span class="c001">% can be used of course with other regularization functions, intercept,...</span><br>
param.regul=<span class="c003">'l1'</span>;<br>
param.loss=<span class="c003">'weighted-logistic'</span>;<br>
param.lambda=0.01;<br>
<span class="c002">fprintf</span>(<span class="c003">'\nFISTA + weighted Logistic l1 + sparse matrix\n'</span>);<br>
<span class="c002">tic</span><br>
[W optim_info]=mexFistaFlat(Y,X,W0,param);<br>
t=<span class="c002">toc</span>;<br>
<span class="c001">% can be used of course with other regularization functions, intercept,...</span><br>
<br>
param.loss=<span class="c003">'logistic'</span>;<br>
<span class="c002">fprintf</span>(<span class="c003">'\nFISTA + Logistic l1 + sparse matrix\n'</span>);<br>
<span class="c002">tic</span><br>
[W optim_info]=mexFistaFlat(Y,<span class="c002">sparse</span>(X),W0,param);<br>
t=<span class="c002">toc</span>;<br>
<span class="c002">fprintf</span>(<span class="c003">'mean loss: %f, mean relative duality_gap: %f, time: %f, number of iterations: %f\n'</span>,<span class="c002">mean</span>(optim_info(1,:)),<span class="c002">mean</span>(optim_info(3,:)),t,<span class="c002">mean</span>(optim_info(4,:)));<br>
<span class="c001">% can be used of course with other regularization functions, intercept,...<br>
<br>
% Multi-Class classification</span><br>
Y=double(<span class="c002">ceil</span>(5*<span class="c002">rand</span>(100,1000))-1);<br>
param.loss=<span class="c003">'multi-logistic'</span>;<br>
<span class="c002">fprintf</span>(<span class="c003">'\nFISTA + Multi-Class Logistic l1\n'</span>);<br>
<span class="c002">tic</span><br>
nclasses=<span class="c002">max</span>(Y(:))+1;<br>
W0=<span class="c002">zeros</span>(<span class="c002">size</span>(X,2),nclasses*<span class="c002">size</span>(Y,2));<br>
[W optim_info]=mexFistaFlat(Y,X,W0,param);<br>
t=<span class="c002">toc</span>;<br>
<span class="c002">fprintf</span>(<span class="c003">'mean loss: %f, mean relative duality_gap: %f, time: %f, number of iterations: %f\n'</span>,<span class="c002">mean</span>(optim_info(1,:)),<span class="c002">mean</span>(optim_info(3,:)),t,<span class="c002">mean</span>(optim_info(4,:)));<br>
<span class="c002">pause</span><br>
<span class="c001">% can be used of course with other regularization functions, intercept,...<br>
<br>
% Multi-Task regression</span><br>
Y=<span class="c002">randn</span>(100,100);<br>
Y=Y-repmat(<span class="c002">mean</span>(Y),[<span class="c002">size</span>(Y,1) 1]);<br>
Y=mexNormalize(Y);<br>
param.compute_gram=false;<br>
W0=<span class="c002">zeros</span>(<span class="c002">size</span>(X,2),<span class="c002">size</span>(Y,2));<br>
param.loss=<span class="c003">'square'</span>;<br>
<span class="c002">fprintf</span>(<span class="c003">'\nFISTA + Regression l1l2 \n'</span>);<br>
param.regul=<span class="c003">'l1l2'</span>;<br>
<span class="c002">tic</span><br>
[W optim_info]=mexFistaFlat(Y,X,W0,param);<br>
<span class="c002">toc<br>
fprintf</span>(<span class="c003">'mean loss: %f, mean relative duality_gap: %f, time: %f, number of iterations: %f\n'</span>,<span class="c002">mean</span>(optim_info(1,:)),<span class="c002">mean</span>(optim_info(3,:)),t,<span class="c002">mean</span>(optim_info(4,:)));<br>
<br>
<span class="c002">fprintf</span>(<span class="c003">'\nFISTA + Regression l1linf \n'</span>);<br>
param.regul=<span class="c003">'l1linf'</span>;<br>
<span class="c002">tic</span><br>
[W optim_info]=mexFistaFlat(Y,X,W0,param);<br>
<span class="c002">toc<br>
fprintf</span>(<span class="c003">'mean loss: %f, mean relative duality_gap: %f, time: %f, number of iterations: %f\n'</span>,<span class="c002">mean</span>(optim_info(1,:)),<span class="c002">mean</span>(optim_info(3,:)),t,<span class="c002">mean</span>(optim_info(4,:)));<br>
<br>
<span class="c002">fprintf</span>(<span class="c003">'\nFISTA + Regression l1l2 + l1 \n'</span>);<br>
param.regul=<span class="c003">'l1l2+l1'</span>;<br>
param.lambda2=0.1;<br>
<span class="c002">tic</span><br>
[W optim_info]=mexFistaFlat(Y,X,W0,param);<br>
<span class="c002">toc<br>
fprintf</span>(<span class="c003">'mean loss: %f, time: %f, number of iterations: %f\n'</span>,<span class="c002">mean</span>(optim_info(1,:)),t,<span class="c002">mean</span>(optim_info(4,:)));<br>
<br>
<span class="c002">fprintf</span>(<span class="c003">'\nFISTA + Regression l1linf + l1 \n'</span>);<br>
param.regul=<span class="c003">'l1linf+l1'</span>;<br>
param.lambda2=0.1;<br>
<span class="c002">tic</span><br>
[W optim_info]=mexFistaFlat(Y,X,W0,param);<br>
<span class="c002">toc<br>
fprintf</span>(<span class="c003">'mean loss: %f, time: %f, number of iterations: %f\n'</span>,<span class="c002">mean</span>(optim_info(1,:)),t,<span class="c002">mean</span>(optim_info(4,:)));<br>
<br>
<span class="c002">fprintf</span>(<span class="c003">'\nFISTA + Regression l1linf + row + columns \n'</span>);<br>
param.regul=<span class="c003">'l1linf-row-column'</span>;<br>
param.lambda2=0.1;<br>
<span class="c002">tic</span><br>
[W optim_info]=mexFistaFlat(Y,X,W0,param);<br>
<span class="c002">toc<br>
fprintf</span>(<span class="c003">'mean loss: %f, mean relative duality_gap: %f, time: %f, number of iterations: %f\n'</span>,<span class="c002">mean</span>(optim_info(1,:)),<span class="c002">mean</span>(optim_info(3,:)),t,<span class="c002">mean</span>(optim_info(4,:)));<br>
<br>
<span class="c001">% Multi-Task Classification</span><br>
<span class="c002">fprintf</span>(<span class="c003">'\nFISTA + Logistic + l1l2 \n'</span>);<br>
param.regul=<span class="c003">'l1l2'</span>;<br>
param.loss=<span class="c003">'logistic'</span>;<br>
Y=2*double(<span class="c002">randn</span>(100,100) &gt; 0)-1;<br>
<span class="c002">tic</span><br>
[W optim_info]=mexFistaFlat(Y,X,W0,param);<br>
<span class="c002">toc<br>
fprintf</span>(<span class="c003">'mean loss: %f, mean relative duality_gap: %f, time: %f, number of iterations: %f\n'</span>,<span class="c002">mean</span>(optim_info(1,:)),<span class="c002">mean</span>(optim_info(3,:)),t,<span class="c002">mean</span>(optim_info(4,:)));<br>
<span class="c001">% Multi-Class + Multi-Task Regularization</span><br>
<br>
<span class="c002">fprintf</span>(<span class="c003">'\nFISTA + Multi-Class Logistic l1l2 \n'</span>);<br>
Y=double(<span class="c002">ceil</span>(5*<span class="c002">rand</span>(100,1000))-1);<br>
param.loss=<span class="c003">'multi-logistic'</span>;<br>
param.regul=<span class="c003">'l1l2'</span>;<br>
<span class="c002">tic</span><br>
nclasses=<span class="c002">max</span>(Y(:))+1;<br>
W0=<span class="c002">zeros</span>(<span class="c002">size</span>(X,2),nclasses*<span class="c002">size</span>(Y,2));<br>
[W optim_info]=mexFistaFlat(Y,X,W0,param);<br>
t=<span class="c002">toc</span>;<br>
<span class="c002">fprintf</span>(<span class="c003">'mean loss: %f, mean relative duality_gap: %f, time: %f, number of iterations: %f\n'</span>,<span class="c002">mean</span>(optim_info(1,:)),<span class="c002">mean</span>(optim_info(3,:)),t,<span class="c002">mean</span>(optim_info(4,:)));<br>
<span class="c001">% can be used of course with other regularization functions, intercept,...</span></td></tr>
</table>
<!--TOC subsection id="sec38" Function mexFistaTree-->
<h3 class="subsection" id="sec38">5.9  Function mexFistaTree</h3><!--SEC END --><p>
Given a matrix <span class="c009">X</span>=[<span class="c009">x</span><sup>1</sup>,…,<span class="c009">x</span><sup><span class="c007">p</span></sup>]<sup><span class="c007">T</span></sup> in ℝ<sup><span class="c007">m</span> × <span class="c007">p</span></sup>, and a matrix <span class="c009">Y</span>=[<span class="c009">y</span><sup>1</sup>,…,<span class="c009">y</span><sup><span class="c007">n</span></sup>], it solves the optimization problems presented in the previous section, with the same regularization functions as mexProximalTree.
see usage details below:</p><table class="lstframe c011"><tr><td class="mouselstlisting"><span class="c001">% <br>
% Usage: [W [optim]]=mexFistaTree(Y,X,W0,tree,param);<br>
%<br>
% Name: mexFistaTree<br>
%<br>
% Description: mexFistaTree solves sparse regularized problems.<br>
%         X is a design matrix of size m x p<br>
%         X=[x^1,...,x^n]', where the x_i's are the rows of X<br>
%         Y=[y^1,...,y^n] is a matrix of size m x n<br>
%         It implements the algorithms FISTA, ISTA and subgradient descent for solving<br>
%<br>
%           min_W  loss(W) + lambda psi(W)<br>
%          <br>
%         The function psi are those used by mexProximalTree (see documentation)<br>
%         for the loss functions, see the documentation of mexFistaFlat<br>
%<br>
%         This function can also handle intercepts (last row of W is not regularized),<br>
%         and/or non-negativity constraints on W and sparse matrices X<br>
%<br>
% Inputs: Y:  double dense m x n matrix<br>
%         X:  double dense or sparse m x p matrix   <br>
%         W0:  double dense p x n matrix or p x Nn matrix (for multi-logistic loss)<br>
%              initial guess<br>
%         tree: struct (see documentation of mexProximalTree)<br>
%         param: struct<br>
%            param.loss (choice of loss, see above)<br>
%            param.regul (choice of regularization, see function mexProximalFlat)<br>
%            param.lambda (regularization parameter)<br>
%            param.lambda2 (optional, regularization parameter, 0 by default)<br>
%            param.lambda3 (optional, regularization parameter, 0 by default)<br>
%            param.verbose (optional, verbosity level, false by default)<br>
%            param.pos (optional, adds positivity constraints on the<br>
%                coefficients, false by default)<br>
%            param.transpose (optional, transpose the matrix in the regularization function)<br>
%            param.size_group (optional, for regularization functions assuming a group<br>
%                 structure)<br>
%            param.numThreads (optional, number of threads for exploiting<br>
%                multi-core / multi-cpus. By default, it takes the value -1,<br>
%                which automatically selects all the available CPUs/cores).<br>
%            param.max_it (optional, maximum number of iterations, 100 by default)<br>
%            param.it0 (optional, frequency for computing duality gap, every 10 iterations by default)<br>
%            param.tol (optional, tolerance for stopping criteration, which is a relative duality gap<br>
%                if it is available, or a relative change of parameters).<br>
%            param.gamma (optional, multiplier for increasing the parameter L in fista, 1.5 by default)<br>
%            param.L0 (optional, initial parameter L in fista, 0.1 by default, should be small enough)<br>
%            param.fixed_step (deactive the line search for L in fista and use param.L0 instead)<br>
%            param.compute_gram (optional, pre-compute X^TX, false by default).<br>
%            param.intercept (optional, do not regularize last row of W, false by default).<br>
%            param.ista (optional, use ista instead of fista, false by default).<br>
%            param.subgrad (optional, if not param.ista, use subradient descent instead of fista, false by default).<br>
%            param.a, param.b (optional, if param.subgrad, the gradient step is a/(t+b)<br>
%            also similar options as mexProximalTree<br>
%<br>
%            the function also implements the ADMM algorithm via an option param.admm=true. It is not documented<br>
%            and you need to look at the source code to use it.<br>
%<br>
% Output:  W:  double dense p x n matrix or p x Nn matrix (for multi-logistic loss)<br>
%          optim: optional, double dense 4 x n matrix.<br>
%              first row: values of the objective functions.<br>
%              third row: values of the relative duality gap (if available)<br>
%              fourth row: number of iterations<br>
%<br>
% Author: Julien Mairal, 2010</span></td></tr>
</table><p>
The following piece of code illustrates how to use this function.
</p><table class="lstframe c011"><tr><td class="mouselstlisting"><span class="c002">format</span> compact;<br>
param.num_threads=-1; <span class="c001">% all cores (-1 by default)</span><br>
param.verbose=false;   <span class="c001">% verbosity, false by default</span><br>
param.lambda=0.001; <span class="c001">% regularization parameter</span><br>
param.it0=10;      <span class="c001">% frequency for duality gap computations</span><br>
param.max_it=200; <span class="c001">% maximum number of iterations</span><br>
param.L0=0.1;<br>
param.tol=1e-5;<br>
param.intercept=false;<br>
param.pos=false;<br>
<br>
<span class="c001">% Example 2 of tree structure<br>
% tree structured groups:<br>
% g1= {0 1 2 3 4 5 6 7 8 9}    root(g1) = { };<br>
% g2= {0 1 2 3 4 5}            root(g2) = {0 1 2};<br>
% g3= {3 4}                    root(g3) = {3 4};<br>
% g4= {5}                      root(g4) = {5};<br>
% g5= {6 7 8 9}                root(g5) = { };<br>
% g6= {6 7}                    root(g6) = {6 7};<br>
% g7= {8 9}                    root(g7) = {8};<br>
% g8 = {9}                     root(g8) = {9};</span><br>
tree.own_variables=  int32([0 0 3 5 6 6 8 9]);   <span class="c001">% pointer to the first variable of each group</span><br>
tree.N_own_variables=int32([0 3 2 1 0 2 1 1]); <span class="c001">% number of "root" variables in each group</span><br>
tree.eta_g=[1 1 1 2 2 2 2.5 2.5];<br>
tree.groups=<span class="c002">sparse</span>([0 0 0 0 0 0 0 0; ...<br>
                    1 0 0 0 0 0 0 0; ...<br>
                    0 1 0 0 0 0 0 0; ...<br>
                    0 1 0 0 0 0 0 0; ...<br>
                    1 0 0 0 0 0 0 0; ...<br>
                    0 0 0 0 1 0 0 0; ...<br>
                    0 0 0 0 1 0 0 0; ...<br>
                    0 0 0 0 0 0 1 0]);  <span class="c001">% first group should always be the root of the tree</span><br>
<br>
X=<span class="c002">randn</span>(100,10);<br>
X=X-repmat(<span class="c002">mean</span>(X),[<span class="c002">size</span>(X,1) 1]);<br>
X=mexNormalize(X);<br>
Y=<span class="c002">randn</span>(100,100);<br>
Y=Y-repmat(<span class="c002">mean</span>(Y),[<span class="c002">size</span>(Y,1) 1]);<br>
Y=mexNormalize(Y);<br>
W0=<span class="c002">zeros</span>(<span class="c002">size</span>(X,2),<span class="c002">size</span>(Y,2));<br>
<span class="c001">% Regression experiments <br>
% 100 regression problems with the same design matrix X.</span><br>
<span class="c002">fprintf</span>(<span class="c003">'\nVarious regression experiments\n'</span>);<br>
param.compute_gram=true;<br>
<span class="c002">fprintf</span>(<span class="c003">'\nFISTA + Regression tree-l2\n'</span>);<br>
param.loss=<span class="c003">'square'</span>;<br>
param.regul=<span class="c003">'tree-l2'</span>;<br>
<span class="c002">tic</span><br>
[W optim_info]=mexFistaTree(Y,X,W0,tree,param);<br>
t=<span class="c002">toc</span>;<br>
<span class="c002">fprintf</span>(<span class="c003">'mean loss: %f, time: %f, number of iterations: %f\n'</span>,<span class="c002">mean</span>(optim_info(1,:)),t,<span class="c002">mean</span>(optim_info(4,:)));<br>
<br>
<span class="c002">fprintf</span>(<span class="c003">'\nFISTA + Regression tree-linf\n'</span>);<br>
param.regul=<span class="c003">'tree-linf'</span>;<br>
<span class="c002">tic</span><br>
[W optim_info]=mexFistaTree(Y,X,W0,tree,param);<br>
t=<span class="c002">toc</span>;<br>
<span class="c002">fprintf</span>(<span class="c003">'mean loss: %f, mean relative duality_gap: %f, time: %f, number of iterations: %f\n'</span>,<span class="c002">mean</span>(optim_info(1,:)),<span class="c002">mean</span>(optim_info(3,:)),t,<span class="c002">mean</span>(optim_info(4,:)));<br>
<br>
<span class="c001">% works also with non tree-structured regularization. tree is ignored</span><br>
<span class="c002">fprintf</span>(<span class="c003">'\nFISTA + Regression Fused-Lasso\n'</span>);<br>
param.regul=<span class="c003">'fused-lasso'</span>;<br>
param.lambda2=0.001;<br>
param.lambda3=0.001; <span class="c001">%</span><br>
<span class="c002">tic</span><br>
[W optim_info]=mexFistaTree(Y,X,W0,tree,param);<br>
t=<span class="c002">toc</span>;<br>
<span class="c002">fprintf</span>(<span class="c003">'mean loss: %f, time: %f, number of iterations: %f\n'</span>,<span class="c002">mean</span>(optim_info(1,:)),t,<span class="c002">mean</span>(optim_info(4,:)));<br>
<br>
<span class="c002">fprintf</span>(<span class="c003">'\nISTA + Regression tree-l0\n'</span>);<br>
param.regul=<span class="c003">'tree-l0'</span>;<br>
<span class="c002">tic</span><br>
[W optim_info]=mexFistaTree(Y,X,W0,tree,param);<br>
t=<span class="c002">toc</span>;<br>
<span class="c002">fprintf</span>(<span class="c003">'mean loss: %f, time: %f, number of iterations: %f\n'</span>,<span class="c002">mean</span>(optim_info(1,:)),t,<span class="c002">mean</span>(optim_info(4,:)));<br>
<br>
<span class="c002">fprintf</span>(<span class="c003">'\nFISTA + Regression tree-l2 with intercept \n'</span>);<br>
param.intercept=true;<br>
param.regul=<span class="c003">'tree-l2'</span>;<br>
<span class="c002">tic</span><br>
[W optim_info]=mexFistaTree(Y,[X ones(<span class="c002">size</span>(X,1),1)],[W0; <span class="c002">zeros</span>(1,<span class="c002">size</span>(W0,2))],tree,param);<br>
t=<span class="c002">toc</span>;<br>
<span class="c002">fprintf</span>(<span class="c003">'mean loss: %f, time: %f, number of iterations: %f\n'</span>,<span class="c002">mean</span>(optim_info(1,:)),t,<span class="c002">mean</span>(optim_info(4,:)));<br>
param.intercept=false;<br>
<br>
<span class="c001">% Classification</span><br>
<span class="c002">fprintf</span>(<span class="c003">'\nOne classification experiment\n'</span>);<br>
Y=2*double(<span class="c002">randn</span>(100,<span class="c002">size</span>(Y,2)) &gt; 0)-1;<br>
<span class="c002">fprintf</span>(<span class="c003">'\nFISTA + Logistic + tree-linf\n'</span>);<br>
param.regul=<span class="c003">'tree-linf'</span>;<br>
param.loss=<span class="c003">'logistic'</span>;<br>
param.lambda=0.001;<br>
<span class="c002">tic</span><br>
[W optim_info]=mexFistaTree(Y,X,W0,tree,param);<br>
t=<span class="c002">toc</span>;<br>
<span class="c002">fprintf</span>(<span class="c003">'mean loss: %f, mean relative duality_gap: %f, time: %f, number of iterations: %f\n'</span>,<span class="c002">mean</span>(optim_info(1,:)),<span class="c002">mean</span>(optim_info(3,:)),t,<span class="c002">mean</span>(optim_info(4,:)));<br>
<span class="c001">% can be used of course with other regularization functions, intercept,...<br>
<br>
% Multi-Class classification</span><br>
Y=double(<span class="c002">ceil</span>(5*<span class="c002">rand</span>(100,<span class="c002">size</span>(Y,2)))-1);<br>
param.loss=<span class="c003">'multi-logistic'</span>;<br>
param.regul=<span class="c003">'tree-l2'</span>;<br>
<span class="c002">fprintf</span>(<span class="c003">'\nFISTA + Multi-Class Logistic + tree-l2 \n'</span>);<br>
<span class="c002">tic</span><br>
nclasses=<span class="c002">max</span>(Y(:))+1;<br>
W0=<span class="c002">zeros</span>(<span class="c002">size</span>(X,2),nclasses*<span class="c002">size</span>(Y,2));<br>
[W optim_info]=mexFistaTree(Y,X,W0,tree,param);<br>
t=<span class="c002">toc</span>;<br>
<span class="c002">fprintf</span>(<span class="c003">'mean loss: %f, time: %f, number of iterations: %f\n'</span>,<span class="c002">mean</span>(optim_info(1,:)),t,<span class="c002">mean</span>(optim_info(4,:)));<br>
<span class="c001">% can be used of course with other regularization functions, intercept,...<br>
<br>
% Multi-Task regression</span><br>
Y=<span class="c002">randn</span>(100,<span class="c002">size</span>(Y,2));<br>
Y=Y-repmat(<span class="c002">mean</span>(Y),[<span class="c002">size</span>(Y,1) 1]);<br>
Y=mexNormalize(Y);<br>
param.compute_gram=false;<br>
param.verbose=true;   <span class="c001">% verbosity, false by default</span><br>
W0=<span class="c002">zeros</span>(<span class="c002">size</span>(X,2),<span class="c002">size</span>(Y,2));<br>
param.loss=<span class="c003">'square'</span>;<br>
<span class="c002">fprintf</span>(<span class="c003">'\nFISTA + Regression multi-task-tree \n'</span>);<br>
param.regul=<span class="c003">'multi-task-tree'</span>;<br>
param.lambda2=0.001;<br>
<span class="c002">tic</span><br>
[W optim_info]=mexFistaTree(Y,X,W0,tree,param);<br>
<span class="c002">toc<br>
fprintf</span>(<span class="c003">'mean loss: %f, mean relative duality_gap: %f, time: %f, number of iterations: %f\n'</span>,<span class="c002">mean</span>(optim_info(1,:)),<span class="c002">mean</span>(optim_info(3,:)),t,<span class="c002">mean</span>(optim_info(4,:)));<br>
<br>
<span class="c001">% Multi-Task Classification</span><br>
<span class="c002">fprintf</span>(<span class="c003">'\nFISTA + Logistic + multi-task-tree \n'</span>);<br>
param.regul=<span class="c003">'multi-task-tree'</span>;<br>
param.lambda2=0.001;<br>
param.loss=<span class="c003">'logistic'</span>;<br>
Y=2*double(<span class="c002">randn</span>(100,<span class="c002">size</span>(Y,2)) &gt; 0)-1;<br>
<span class="c002">tic</span><br>
[W optim_info]=mexFistaTree(Y,X,W0,tree,param);<br>
<span class="c002">toc<br>
fprintf</span>(<span class="c003">'mean loss: %f, mean relative duality_gap: %f, time: %f, number of iterations: %f\n'</span>,<span class="c002">mean</span>(optim_info(1,:)),<span class="c002">mean</span>(optim_info(3,:)),t,<span class="c002">mean</span>(optim_info(4,:)));<br>
<br>
<span class="c001">% Multi-Class + Multi-Task Regularization</span><br>
param.verbose=false;<br>
<span class="c002">fprintf</span>(<span class="c003">'\nFISTA + Multi-Class Logistic +multi-task-tree \n'</span>);<br>
Y=double(<span class="c002">ceil</span>(5*<span class="c002">rand</span>(100,<span class="c002">size</span>(Y,2)))-1);<br>
param.loss=<span class="c003">'multi-logistic'</span>;<br>
param.regul=<span class="c003">'multi-task-tree'</span>;<br>
<span class="c002">tic</span><br>
nclasses=<span class="c002">max</span>(Y(:))+1;<br>
W0=<span class="c002">zeros</span>(<span class="c002">size</span>(X,2),nclasses*<span class="c002">size</span>(Y,2));<br>
[W optim_info]=mexFistaTree(Y,X,W0,tree,param);<br>
t=<span class="c002">toc</span>;<br>
<span class="c002">fprintf</span>(<span class="c003">'mean loss: %f, mean relative duality_gap: %f, time: %f, number of iterations: %f\n'</span>,<span class="c002">mean</span>(optim_info(1,:)),<span class="c002">mean</span>(optim_info(3,:)),t,<span class="c002">mean</span>(optim_info(4,:)));<br>
<span class="c001">% can be used of course with other regularization functions, intercept,...</span><br>
<br>
<span class="c002">fprintf</span>(<span class="c003">'\nFISTA + Multi-Class Logistic +multi-task-tree + sparse matrix \n'</span>);<br>
<span class="c002">tic</span><br>
nclasses=<span class="c002">max</span>(Y(:))+1;<br>
W0=<span class="c002">zeros</span>(<span class="c002">size</span>(X,2),nclasses*<span class="c002">size</span>(Y,2));<br>
[W optim_info]=mexFistaTree(Y,<span class="c002">sparse</span>(X),W0,tree,param);<br>
t=<span class="c002">toc</span>;<br>
<span class="c002">fprintf</span>(<span class="c003">'mean loss: %f, mean relative duality_gap: %f, time: %f, number of iterations: %f\n'</span>,<span class="c002">mean</span>(optim_info(1,:)),<span class="c002">mean</span>(optim_info(3,:)),t,<span class="c002">mean</span>(optim_info(4,:)));</td></tr>
</table>
<!--TOC subsection id="sec39" Function mexFistaGraph-->
<h3 class="subsection" id="sec39">5.10  Function mexFistaGraph</h3><!--SEC END --><p>
Given a matrix <span class="c009">X</span>=[<span class="c009">x</span><sup>1</sup>,…,<span class="c009">x</span><sup><span class="c007">p</span></sup>]<sup><span class="c007">T</span></sup> in ℝ<sup><span class="c007">m</span> × <span class="c007">p</span></sup>, and a matrix <span class="c009">Y</span>=[<span class="c009">y</span><sup>1</sup>,…,<span class="c009">y</span><sup><span class="c007">n</span></sup>], it solves the optimization problems presented in the previous section, with the same regularization functions as mexProximalGraph.
see usage details below:</p><table class="lstframe c011"><tr><td class="mouselstlisting"><span class="c001">% <br>
% Usage: [W [optim]]=mexFistaGraph(Y,X,W0,graph,param);<br>
%<br>
% Name: mexFistaGraph<br>
%<br>
% Description: mexFistaGraph solves sparse regularized problems.<br>
%         X is a design matrix of size m x p<br>
%         X=[x^1,...,x^n]', where the x_i's are the rows of X<br>
%         Y=[y^1,...,y^n] is a matrix of size m x n<br>
%         It implements the algorithms FISTA, ISTA and subgradient descent.<br>
%<br>
%         It implements the algorithms FISTA, ISTA and subgradient descent for solving<br>
%<br>
%           min_W  loss(W) + lambda psi(W)<br>
%          <br>
%         The function psi are those used by mexProximalGraph (see documentation)<br>
%         for the loss functions, see the documentation of mexFistaFlat<br>
%         <br>
%         This function can also handle intercepts (last row of W is not regularized),<br>
%         and/or non-negativity constraints on W.<br>
%<br>
% Inputs: Y:  double dense m x n matrix<br>
%         X:  double dense or sparse m x p matrix   <br>
%         W0:  double dense p x n matrix or p x Nn matrix (for multi-logistic loss)<br>
%              initial guess<br>
%         graph: struct (see documentation of mexProximalGraph)<br>
%         param: struct<br>
%            param.loss (choice of loss, see above)<br>
%            param.regul (choice of regularization, see function mexProximalFlat)<br>
%            param.lambda (regularization parameter)<br>
%            param.lambda2 (optional, regularization parameter, 0 by default)<br>
%            param.lambda3 (optional, regularization parameter, 0 by default)<br>
%            param.verbose (optional, verbosity level, false by default)<br>
%            param.pos (optional, adds positivity constraints on the<br>
%                coefficients, false by default)<br>
%            param.numThreads (optional, number of threads for exploiting<br>
%                multi-core / multi-cpus. By default, it takes the value -1,<br>
%                which automatically selects all the available CPUs/cores).<br>
%            param.max_it (optional, maximum number of iterations, 100 by default)<br>
%            param.it0 (optional, frequency for computing duality gap, every 10 iterations by default)<br>
%            param.tol (optional, tolerance for stopping criteration, which is a relative duality gap<br>
%                if it is available, or a relative change of parameters).<br>
%            param.gamma (optional, multiplier for increasing the parameter L in fista, 1.5 by default)<br>
%            param.L0 (optional, initial parameter L in fista, 0.1 by default, should be small enough)<br>
%            param.fixed_step (deactive the line search for L in fista and use param.L0 instead)<br>
%            param.compute_gram (optional, pre-compute X^TX, false by default).<br>
%            param.intercept (optional, do not regularize last row of W, false by default).<br>
%            param.ista (optional, use ista instead of fista, false by default).<br>
%            param.subgrad (optional, if not param.ista, use subradient descent instead of fista, false by default).<br>
%            param.a, param.b (optional, if param.subgrad, the gradient step is a/(t+b)<br>
%            also similar options as mexProximalTree<br>
%<br>
%            the function also implements the ADMM algorithm via an option param.admm=true. It is not documented<br>
%            and you need to look at the source code to use it.<br>
%<br>
%<br>
% Output:  W:  double dense p x n matrix or p x Nn matrix (for multi-logistic loss)<br>
%          optim: optional, double dense 4 x n matrix.<br>
%              first row: values of the objective functions.<br>
%              third row: values of the relative duality gap (if available)<br>
%              fourth row: number of iterations<br>
%<br>
% Author: Julien Mairal, 2010</span></td></tr>
</table><p>
The following piece of code illustrates how to use this function.
</p><table class="lstframe c011"><tr><td class="mouselstlisting"><span class="c002">clear</span> <span class="c002">all</span>;<br>
<span class="c002">randn</span>(<span class="c003">'seed'</span>,0);<br>
<span class="c002">format</span> compact;<br>
param.num_threads=-1; <span class="c001">% all cores (-1 by default)</span><br>
param.verbose=false;   <span class="c001">% verbosity, false by default</span><br>
param.lambda=0.1; <span class="c001">% regularization parameter</span><br>
param.it0=1;      <span class="c001">% frequency for duality gap computations</span><br>
param.max_it=100; <span class="c001">% maximum number of iterations</span><br>
param.L0=0.1;<br>
param.tol=1e-5;<br>
param.intercept=false;<br>
param.pos=false;<br>
<br>
graph.eta_g=[1 1 1 1 1];<br>
graph.groups=<span class="c002">sparse</span>([0 0 0 1 0;<br>
                     0 0 0 0 0;<br>
                     0 0 0 0 0;<br>
                     0 0 0 0 0;<br>
                     0 0 1 0 0]);   <span class="c001">% g5 is included in g3, and g2 is included in g4</span><br>
graph.groups_var=<span class="c002">sparse</span>([1 0 0 0 0;<br>
                         1 0 0 0 0;<br>
                         1 0 0 0 0 ;<br>
                         1 1 0 0 0;<br>
                         0 1 0 1 0;<br>
                         0 1 0 1 0;<br>
                         0 1 0 0 1;<br>
                         0 0 0 0 1;<br>
                         0 0 0 0 1;<br>
                         0 0 1 0 0]); <span class="c001">% represents direct inclusion relations </span><br>
<br>
X=<span class="c002">randn</span>(100,10);<br>
param.verbose=true;<br>
<span class="c001">%X=eye(10);</span><br>
X=X-repmat(<span class="c002">mean</span>(X),[<span class="c002">size</span>(X,1) 1]);<br>
X=mexNormalize(X);<br>
Y=<span class="c002">randn</span>(100,1);<br>
Y=Y-repmat(<span class="c002">mean</span>(Y),[<span class="c002">size</span>(Y,1) 1]);<br>
Y=mexNormalize(Y);<br>
W0=<span class="c002">zeros</span>(<span class="c002">size</span>(X,2),<span class="c002">size</span>(Y,2));<br>
<span class="c001">% Regression experiments <br>
% 100 regression problems with the same design matrix X.</span><br>
<span class="c002">fprintf</span>(<span class="c003">'\nVarious regression experiments\n'</span>);<br>
param.compute_gram=true;<br>
<span class="c002">fprintf</span>(<span class="c003">'\nFISTA + Regression graph\n'</span>);<br>
param.loss=<span class="c003">'square'</span>;<br>
param.regul=<span class="c003">'graph'</span>;<br>
<span class="c002">tic</span><br>
[W optim_info]=mexFistaGraph(Y,X,W0,graph,param);<br>
t=<span class="c002">toc</span>;<br>
<span class="c002">fprintf</span>(<span class="c003">'mean loss: %f, mean relative duality_gap: %f, time: %f, number of iterations: %f\n'</span>,<span class="c002">mean</span>(optim_info(1,:)),<span class="c002">mean</span>(optim_info(3,:)),t,<span class="c002">mean</span>(optim_info(4,:)));<br>
<br>
<span class="c002">fprintf</span>(<span class="c003">'\nADMM + Regression graph\n'</span>);<br>
param.admm=true;<br>
param.lin_admm=true;<br>
param.c=1;<br>
param.delta=1;<br>
<span class="c002">tic</span><br>
[W optim_info]=mexFistaGraph(Y,X,W0,graph,param);<br>
t=<span class="c002">toc</span>;<br>
<span class="c002">fprintf</span>(<span class="c003">'mean loss: %f, stopping criterion: %f, time: %f, number of iterations: %f\n'</span>,<span class="c002">mean</span>(optim_info(1,:)),<span class="c002">mean</span>(optim_info(3,:)),t,<span class="c002">mean</span>(optim_info(4,:)));<br>
<br>
param.admm=false;<br>
param.max_it=5;<br>
param.it0=1;<br>
<span class="c002">tic</span><br>
[W optim_info]=mexFistaGraph(Y,X,W,graph,param);<br>
t=<span class="c002">toc</span>;<br>
<span class="c002">fprintf</span>(<span class="c003">'mean loss: %f, mean relative duality_gap: %f, time: %f, number of iterations: %f\n'</span>,<span class="c002">mean</span>(optim_info(1,:)),<span class="c002">mean</span>(optim_info(3,:)),t,<span class="c002">mean</span>(optim_info(4,:)));<br>
<br>
<span class="c001">% works also with non graph-structured regularization. graph is ignored</span><br>
<span class="c002">fprintf</span>(<span class="c003">'\nFISTA + Regression Fused-Lasso\n'</span>);<br>
param.regul=<span class="c003">'fused-lasso'</span>;<br>
param.lambda2=0.01;<br>
param.lambda3=0.01; <span class="c001">%</span><br>
<span class="c002">tic</span><br>
[W optim_info]=mexFistaGraph(Y,X,W0,graph,param);<br>
t=<span class="c002">toc</span>;<br>
<span class="c002">fprintf</span>(<span class="c003">'mean loss: %f, time: %f, number of iterations: %f\n'</span>,<span class="c002">mean</span>(optim_info(1,:)),t,<span class="c002">mean</span>(optim_info(4,:)));<br>
<br>
<span class="c002">fprintf</span>(<span class="c003">'\nFISTA + Regression graph with intercept \n'</span>);<br>
param.intercept=true;<br>
param.regul=<span class="c003">'graph'</span>;<br>
<span class="c002">tic</span><br>
[W optim_info]=mexFistaGraph(Y,[X ones(<span class="c002">size</span>(X,1),1)],[W0; <span class="c002">zeros</span>(1,<span class="c002">size</span>(W0,2))],graph,param);<br>
t=<span class="c002">toc</span>;<br>
<span class="c002">fprintf</span>(<span class="c003">'mean loss: %f, mean relative duality_gap: %f, time: %f, number of iterations: %f\n'</span>,<span class="c002">mean</span>(optim_info(1,:)),<span class="c002">mean</span>(optim_info(3,:)),t,<span class="c002">mean</span>(optim_info(4,:)));<br>
param.intercept=false;<br>
<br>
<span class="c001">% Classification</span><br>
<span class="c002">fprintf</span>(<span class="c003">'\nOne classification experiment\n'</span>);<br>
Y=2*double(<span class="c002">randn</span>(100,<span class="c002">size</span>(Y,2)) &gt; 0)-1;<br>
<span class="c002">fprintf</span>(<span class="c003">'\nFISTA + Logistic + graph-linf\n'</span>);<br>
param.regul=<span class="c003">'graph'</span>;<br>
param.loss=<span class="c003">'logistic'</span>;<br>
param.lambda=0.01;<br>
<span class="c002">tic</span><br>
[W optim_info]=mexFistaGraph(Y,X,W0,graph,param);<br>
t=<span class="c002">toc</span>;<br>
<span class="c002">fprintf</span>(<span class="c003">'mean loss: %f, mean relative duality_gap: %f, time: %f, number of iterations: %f\n'</span>,<span class="c002">mean</span>(optim_info(1,:)),<span class="c002">mean</span>(optim_info(3,:)),t,<span class="c002">mean</span>(optim_info(4,:)));<br>
<span class="c001">% can be used of course with other regularization functions, intercept,...<br>
<br>
% Multi-Class classification</span><br>
Y=double(<span class="c002">ceil</span>(5*<span class="c002">rand</span>(100,<span class="c002">size</span>(Y,2)))-1);<br>
param.loss=<span class="c003">'multi-logistic'</span>;<br>
param.regul=<span class="c003">'graph'</span>;<br>
<span class="c002">fprintf</span>(<span class="c003">'\nFISTA + Multi-Class Logistic + graph \n'</span>);<br>
<span class="c002">tic</span><br>
nclasses=<span class="c002">max</span>(Y(:))+1;<br>
W0=<span class="c002">zeros</span>(<span class="c002">size</span>(X,2),nclasses*<span class="c002">size</span>(Y,2));<br>
[W optim_info]=mexFistaGraph(Y,X,W0,graph,param);<br>
t=<span class="c002">toc</span>;<br>
<span class="c002">fprintf</span>(<span class="c003">'mean loss: %f, mean relative duality_gap: %f, time: %f, number of iterations: %f\n'</span>,<span class="c002">mean</span>(optim_info(1,:)),<span class="c002">mean</span>(optim_info(3,:)),t,<span class="c002">mean</span>(optim_info(4,:)));<br>
<span class="c001">% can be used of course with other regularization functions, intercept,...<br>
<br>
% Multi-Task regression</span><br>
Y=<span class="c002">randn</span>(100,<span class="c002">size</span>(Y,2));<br>
Y=Y-repmat(<span class="c002">mean</span>(Y),[<span class="c002">size</span>(Y,1) 1]);<br>
Y=mexNormalize(Y);<br>
param.compute_gram=false;<br>
param.verbose=true;   <span class="c001">% verbosity, false by default</span><br>
W0=<span class="c002">zeros</span>(<span class="c002">size</span>(X,2),<span class="c002">size</span>(Y,2));<br>
param.loss=<span class="c003">'square'</span>;<br>
<span class="c002">fprintf</span>(<span class="c003">'\nFISTA + Regression multi-task-graph \n'</span>);<br>
param.regul=<span class="c003">'multi-task-graph'</span>;<br>
param.lambda2=0.01;<br>
<span class="c002">tic</span><br>
[W optim_info]=mexFistaGraph(Y,X,W0,graph,param);<br>
<span class="c002">toc<br>
fprintf</span>(<span class="c003">'mean loss: %f, mean relative duality_gap: %f, time: %f, number of iterations: %f\n'</span>,<span class="c002">mean</span>(optim_info(1,:)),<span class="c002">mean</span>(optim_info(3,:)),t,<span class="c002">mean</span>(optim_info(4,:)));<br>
<br>
<span class="c001">% Multi-Task Classification</span><br>
<span class="c002">fprintf</span>(<span class="c003">'\nFISTA + Logistic + multi-task-graph \n'</span>);<br>
param.regul=<span class="c003">'multi-task-graph'</span>;<br>
param.lambda2=0.01;<br>
param.loss=<span class="c003">'logistic'</span>;<br>
Y=2*double(<span class="c002">randn</span>(100,<span class="c002">size</span>(Y,2)) &gt; 0)-1;<br>
<span class="c002">tic</span><br>
[W optim_info]=mexFistaGraph(Y,X,W0,graph,param);<br>
<span class="c002">toc<br>
fprintf</span>(<span class="c003">'mean loss: %f, mean relative duality_gap: %f, time: %f, number of iterations: %f\n'</span>,<span class="c002">mean</span>(optim_info(1,:)),<span class="c002">mean</span>(optim_info(3,:)),t,<span class="c002">mean</span>(optim_info(4,:)));<br>
<span class="c001">% Multi-Class + Multi-Task Regularization</span><br>
<br>
param.verbose=false;<br>
<span class="c002">fprintf</span>(<span class="c003">'\nFISTA + Multi-Class Logistic +multi-task-graph \n'</span>);<br>
Y=double(<span class="c002">ceil</span>(5*<span class="c002">rand</span>(100,<span class="c002">size</span>(Y,2)))-1);<br>
param.loss=<span class="c003">'multi-logistic'</span>;<br>
param.regul=<span class="c003">'multi-task-graph'</span>;<br>
<span class="c002">tic</span><br>
nclasses=<span class="c002">max</span>(Y(:))+1;<br>
W0=<span class="c002">zeros</span>(<span class="c002">size</span>(X,2),nclasses*<span class="c002">size</span>(Y,2));<br>
[W optim_info]=mexFistaGraph(Y,X,W0,graph,param);<br>
t=<span class="c002">toc</span>;<br>
<span class="c002">fprintf</span>(<span class="c003">'mean loss: %f, mean relative duality_gap: %f, time: %f, number of iterations: %f\n'</span>,<span class="c002">mean</span>(optim_info(1,:)),<span class="c002">mean</span>(optim_info(3,:)),t,<span class="c002">mean</span>(optim_info(4,:)));<br>
<span class="c001">% can be used of course with other regularization functions, intercept,...</span></td></tr>
</table>
<!--TOC subsection id="sec40" Function mexFistaPathCoding-->
<h3 class="subsection" id="sec40">5.11  Function mexFistaPathCoding</h3><!--SEC END --><p>
Similarly, the toolbox handles the penalties of [<a href="#mairal14">24</a>] with the following function
</p><table class="lstframe c011"><tr><td class="mouselstlisting"><span class="c001">% <br>
% Usage: [W [optim]]=mexFistaPathCoding(Y,X,W0,DAG,param);<br>
%<br>
% Name: mexFistaPathCoding<br>
%<br>
% Description: mexFistaPathCoding solves sparse regularized problems for the <br>
%         path coding penalties of http://arxiv.org/abs/1204.4539<br>
%         X is a design matrix of size m x p<br>
%         X=[x^1,...,x^n]', where the x_i's are the rows of X<br>
%         Y=[y^1,...,y^n] is a matrix of size m x n<br>
%         It implements the algorithms FISTA, ISTA and subgradient descent.<br>
%<br>
%         It implements the algorithms FISTA, ISTA and subgradient descent for solving<br>
%<br>
%           min_W  loss(W) + lambda psi(W)<br>
%          <br>
%         The function psi are those used by mexProximalPathCoding (see documentation)<br>
%         for the loss functions, see the documentation of mexFistaFlat<br>
%         <br>
%         This function can also handle intercepts (last row of W is not regularized),<br>
%         and/or non-negativity constraints on W.<br>
%<br>
% Inputs: Y:  double dense m x n matrix<br>
%         X:  double dense or sparse m x p matrix   <br>
%         W0:  double dense p x n matrix or p x Nn matrix (for multi-logistic loss)<br>
%              initial guess<br>
%         DAG: struct (see documentation of mexProximalPathCoding)<br>
%         param: struct<br>
%            param.loss (choice of loss, see above)<br>
%            param.regul (choice of regularization, see function mexProximalPathCoding)<br>
%            param.lambda (regularization parameter)<br>
%            param.lambda2 (optional, regularization parameter, 0 by default)<br>
%            param.lambda3 (optional, regularization parameter, 0 by default)<br>
%            param.verbose (optional, verbosity level, false by default)<br>
%            param.pos (optional, adds positivity constraints on the<br>
%                coefficients, false by default)<br>
%            param.numThreads (optional, number of threads for exploiting<br>
%                multi-core / multi-cpus. By default, it takes the value -1,<br>
%                which automatically selects all the available CPUs/cores).<br>
%            param.max_it (optional, maximum number of iterations, 100 by default)<br>
%            param.it0 (optional, frequency for computing duality gap, every 10 iterations by default)<br>
%            param.tol (optional, tolerance for stopping criteration, which is a relative duality gap<br>
%                if it is available, or a relative change of parameters).<br>
%            param.gamma (optional, multiplier for increasing the parameter L in fista, 1.5 by default)<br>
%            param.L0 (optional, initial parameter L in fista, 0.1 by default, should be small enough)<br>
%            param.fixed_step (deactive the line search for L in fista and use param.L0 instead)<br>
%            param.compute_gram (optional, pre-compute X^TX, false by default).<br>
%            param.intercept (optional, do not regularize last row of W, false by default).<br>
%            param.ista (optional, use ista instead of fista, false by default).<br>
%            param.subgrad (optional, if not param.ista, use subradient descent instead of fista, false by default).<br>
%            param.a, param.b (optional, if param.subgrad, the gradient step is a/(t+b)<br>
%            also similar options as mexProximalPathCoding<br>
%<br>
%<br>
% Output:  W:  double dense p x n matrix or p x Nn matrix (for multi-logistic loss)<br>
%          optim: optional, double dense 4 x n matrix.<br>
%              first row: values of the objective functions.<br>
%              third row: values of the relative duality gap (if available)<br>
%              fourth row: number of iterations<br>
%<br>
% Author: Julien Mairal, 2012</span></td></tr>
</table><p>
The following piece of code illustrates how to use this function.
</p><table class="lstframe c011"><tr><td class="mouselstlisting"><span class="c002">clear</span> <span class="c002">all</span>;<br>
<span class="c002">rand</span>(<span class="c003">'seed'</span>,0);<br>
<span class="c002">randn</span>(<span class="c003">'seed'</span>,0);<br>
<span class="c002">fprintf</span>(<span class="c003">'test mexFistaPathCoding\n'</span>);<br>
p=100;<br>
n=1000;<br>
<span class="c001">% generate a DAG</span><br>
G=sprand(p,p,0.05);<br>
G=mexRemoveCyclesGraph(G);<br>
<span class="c002">fprintf</span>(<span class="c003">'\n'</span>);<br>
<br>
<span class="c001">% generate a data matrix</span><br>
X=<span class="c002">randn</span>(n,p);<br>
X=X-repmat(<span class="c002">mean</span>(X),[<span class="c002">size</span>(X,1) 1]);<br>
X=mexNormalize(X);<br>
Y=<span class="c002">randn</span>(n,2);<br>
Y=Y-repmat(<span class="c002">mean</span>(Y),[<span class="c002">size</span>(Y,1) 1]);<br>
Y=mexNormalize(Y);<br>
W0=<span class="c002">zeros</span>(<span class="c002">size</span>(X,2),<span class="c002">size</span>(Y,2));<br>
<br>
<span class="c001">% input graph</span><br>
graph.weights=G;<br>
graph.stop_weights=<span class="c002">zeros</span>(1,p);<br>
graph.start_weights=10*ones(1,p);<br>
<br>
<span class="c001">% FISTA parameters</span><br>
param.num_threads=-1; <span class="c001">% all cores (-1 by default)</span><br>
param.verbose=true;   <span class="c001">% verbosity, false by default</span><br>
param.lambda=0.005; <span class="c001">% regularization parameter</span><br>
param.it0=1;      <span class="c001">% frequency for duality gap computations</span><br>
param.max_it=100; <span class="c001">% maximum number of iterations</span><br>
param.L0=0.01;<br>
param.tol=1e-4;<br>
param.precision=10000000;<br>
param.pos=false;<br>
<br>
<span class="c002">fprintf</span>(<span class="c003">'Square Loss + convex path penalty\n'</span>);<br>
param.loss=<span class="c003">'square'</span>;<br>
param.regul=<span class="c003">'graph-path-conv'</span>;<br>
<span class="c002">tic</span><br>
[W1 optim_info]=mexFistaPathCoding(Y,X,W0,graph,param);<br>
t=<span class="c002">toc</span>;<br>
<span class="c002">fprintf</span>(<span class="c003">'mean loss: %f, mean relative duality_gap: %f, time: %f, number of iterations: %f\n'</span>,<span class="c002">mean</span>(optim_info(1,:)),<span class="c002">mean</span>(optim_info(3,:)),t,<span class="c002">mean</span>(optim_info(4,:)));<br>
num=mexCountConnexComponents(graph.weights,W1(:,1));<br>
<span class="c002">fprintf</span>(<span class="c003">'Num of connected components: %d\n'</span>,num);<br>
<br>
<span class="c002">fprintf</span>(<span class="c003">'\n'</span>);<br>
<span class="c002">fprintf</span>(<span class="c003">'Square Loss + non-convex path penalty\n'</span>);<br>
param.loss=<span class="c003">'square'</span>;<br>
param.regul=<span class="c003">'graph-path-l0'</span>;<br>
param.lambda=0.0001; <span class="c001">% regularization parameter</span><br>
param.ista=true;<br>
<span class="c002">tic</span><br>
[W2 optim_info]=mexFistaPathCoding(Y,X,W0,graph,param);<br>
t=<span class="c002">toc</span>;<br>
num=mexCountConnexComponents(graph.weights,W2(:,1));<br>
<span class="c002">fprintf</span>(<span class="c003">'Num of connected components: %d\n'</span>,num);<br>
<br>
<span class="c002">fprintf</span>(<span class="c003">'\n'</span>);<br>
<span class="c002">fprintf</span>(<span class="c003">'Note that for non-convex penalties, continuation strategies sometimes perform better:\n'</span>);<br>
tablambda=param.lambda*<span class="c002">sqrt</span>(<span class="c002">sqrt</span>(<span class="c002">sqrt</span>(2))).^(20:-1:0);<br>
lambda_orig=param.lambda;<br>
<span class="c002">tic</span><br>
W2=W0;<br>
<span class="c002">for</span> ii = 1:<span class="c002">length</span>(tablambda)<br>
   param.lambda=tablambda(ii);<br>
   param.verbose=false;<br>
   [W2]=mexFistaPathCoding(Y,X,W2,graph,param);<br>
<span class="c002">end</span><br>
param.verbose=true;<br>
param.lambda=lambda_orig;<br>
[W2 optim_info]=mexFistaPathCoding(Y,X,W2,graph,param);<br>
t=<span class="c002">toc</span>;<br>
num=mexCountConnexComponents(graph.weights,W2(:,1));<br>
param.ista=false;<br>
<span class="c002">fprintf</span>(<span class="c003">'Num of connected components: %d\n'</span>,num);</td></tr>
</table>
<!--TOC subsection id="sec41" Function solverPoisson-->
<h3 class="subsection" id="sec41">5.12  Function solverPoisson</h3><!--SEC END --><p>
The following problem is addressed here
</p><table class="display dcenter"><tr class="c016"><td class="dcell">   </td><td class="dcell"><table class="display"><tr><td class="dcell c012">&nbsp;</td></tr>
<tr><td class="dcell c012">min</td></tr>
<tr><td class="dcell c012"><span class="c009">w</span> ∈ ℝ<sub>+</sub><sup><span class="c007">p</span></sup></td></tr>
</table></td><td class="dcell">  </td><td class="dcell"><table class="display"><tr><td class="dcell c012"><span class="c007">p</span></td></tr>
<tr><td class="dcell c012"><span class="c006">∑</span></td></tr>
<tr><td class="dcell c012"><span class="c007">j</span>=1</td></tr>
</table></td><td class="dcell"> <span class="c009">x</span><sub><span class="c007">i</span></sub><sup>⊤</sup><span class="c009">w</span> + δ − <span class="c007">y</span><sub><span class="c007">i</span></sub> log(<span class="c009">x</span><sub><span class="c007">i</span></sub><sup>⊤</sup><span class="c009">w</span> +δ) + λ ψ(<span class="c009">w</span>).
</td></tr>
</table><table class="lstframe c011"><tr><td class="mouselstlisting"><span class="c001">% <br>
% Usage:   [W, [optim]]=solverPoisson(Y,X,W0,param);<br>
%<br>
% Name: solverPoisson<br>
%<br>
% Description: solverPoisson solves the regularized Poisson regression<br>
% problem for every column of Y:<br>
%<br>
%   min_w  \sum_i (x_i' w) + delta  - y_i log( x_i' w + delta) + lambda psi(w) (1)<br>
%<br>
% delta should be positive. The method solves (1) for decreasing values of delta,<br>
% until it reaches the desired target value. The algorithm ISTA is used<br>
% with Barzilai-Borwein steps.<br>
%<br>
% Inputs: Y:  double m x n matrix (non-negative values)<br>
%         X:  double m x p matrix (non-negative values)<br>
%         W0:  double p x n matrix (non-negative values)<br>
%         param: struct<br>
%            param.tol, param.max_it, param.verbose, param.L0,<br>
%            param.it0, param.lambda as for mexFistaFlat<br>
%            param.regul,  as for mexProximalFlat<br>
%<br>
% Output:<br>
%         W: double p x n matrix (non-negative values)</span><br>
<br>
<span class="c002">function</span> [W optim] = solverPoisson(Y,X,W0,param)<br>
param.ista=true;<br>
param.intercept=false;<br>
param.pos=true;<br>
param.linesearch_mode=2;<br>
param.loss=<span class="c003">'poisson'</span>;<br>
tabdelta=<span class="c002">logspace</span>(0,<span class="c002">log10</span>(param.delta),-<span class="c002">log10</span>(param.delta));<br>
<span class="c002">for</span> delta = tabdelta<br>
   param2=param;<br>
   param2.delta=delta;<br>
   [W optim]=mexFistaFlat(Y,X,W0,param2);<br>
   W0=W;<br>
<span class="c002">end</span></td></tr>
</table><p>
The following piece of code illustrates how to use this function.
</p><table class="lstframe c011"><tr><td class="mouselstlisting"><span class="c002">rand</span>(<span class="c003">'seed'</span>,0);<br>
<span class="c002">randn</span>(<span class="c003">'seed'</span>,0);<br>
<span class="c002">format</span> compact;<br>
<span class="c002">randn</span>(<span class="c003">'seed'</span>,0);<br>
param.numThreads=-1; <span class="c001">% all cores (-1 by default)</span><br>
param.verbose=true;   <span class="c001">% verbosity, false by default</span><br>
param.lambda=0.05; <span class="c001">% regularization parameter</span><br>
param.it0=10;      <span class="c001">% frequency for duality gap computations</span><br>
param.max_it=200; <span class="c001">% maximum number of iterations</span><br>
param.L0=0.01;<br>
param.tol=1e-6;<br>
param.delta=1e-5;<br>
<br>
X=<span class="c002">rand</span>(1000,2000);<br>
X=mexNormalize(X);<br>
Y=<span class="c002">rand</span>(1000,1);<br>
Y=mexNormalize(Y);<br>
W0=<span class="c002">zeros</span>(<span class="c002">size</span>(X,2),<span class="c002">size</span>(Y,2));<br>
<span class="c001">% Regression experiments <br>
% 100 regression problems with the same design matrix X.</span><br>
<span class="c002">fprintf</span>(<span class="c003">'\nVarious regression experiments\n'</span>);<br>
<span class="c002">fprintf</span>(<span class="c003">'\nFISTA + Regression l1\n'</span>);<br>
param.regul=<span class="c003">'l1'</span>;<br>
<span class="c002">tic</span><br>
[W optim_info]=solverPoisson(Y,X,W0,param);<br>
t=<span class="c002">toc</span>;<br>
<span class="c002">fprintf</span>(<span class="c003">'mean loss: %f, mean relative duality_gap: %f, time: %f, number of iterations: %f\n'</span>,<span class="c002">mean</span>(optim_info(1,:)),<span class="c002">mean</span>(optim_info(3,:)),t,<span class="c002">mean</span>(optim_info(4,:)));</td></tr>
</table>
<!--TOC subsection id="sec42" Function mexIncrementalProx-->
<h3 class="subsection" id="sec42">5.13  Function mexIncrementalProx</h3><!--SEC END --><p>
This implements the incremental solver MISO [<a href="#mairal16">25</a>].
</p><table class="lstframe c011"><tr><td class="mouselstlisting"><span class="c001">% <br>
% Usage:  [W [optim]]=mexIncrementalProx(y,X,W0,param);<br>
%<br>
% Name: mexIncrementalProx<br>
%<br>
% Description: mexIncremrentalProx implements the incremental algorithm MISO<br>
% for composite optimization in a large scale setting.<br>
%        X is a design matrix of size p x n<br>
%        y is a vector of size n <br>
% WARNING, X is transposed compared to the functions mexFista*, and y is a vector<br>
%        param.lambda is a vector that contains nlambda different values of the<br>
%        regularization parameter (it can be a scalar, in that case nlambda=1)<br>
%        W0: is a dense matrix of size p x nlambda   It is in fact ineffective<br>
%        in the current release<br>
%        W: is the output, dense matrix of size p x nlambda<br>
%<br>
%         - if param.loss='square' and param.regul corresponds to a regularization<br>
%           function (currently 'l1' or 'l2'), the following problem is solved<br>
%           w = argmin (1/n)sum_{i=1}^n 0.5(y_i- x_i^T w)^2 + lambda psi(w)<br>
%         - if param.loss='logistic' and param.regul corresponds to a regularization<br>
%           function (currently 'l1' or 'l2'), the following problem is solved<br>
%           w = argmin (1/n)sum_{i=1}^n log(1+ exp(-y_ix_i^T w)) + lambda psi(w)<br>
%           Note that here, the y_i's should be -1 or +1 <br>
%          <br>
%         The current release does not handle intercepts<br>
%<br>
% Inputs: y: double dense vector of size n<br>
%         X: dense or sparse matrix of size p x n<br>
%         W0: dense matrix of size p x nlambda<br>
%         param: struct<br>
%           param.loss (choice of loss)<br>
%           param.regul (choice of regularization function)<br>
%           param.lambda : vector of size nlambda<br>
%           param.epochs: number of passes over the data<br>
%           param.minibatches: size of the mini-batches: recommended value is 1<br>
%              if X is dense, and min(n,ceil(1/density)) if X is sparse<br>
%           param.warm_restart : (path-following strategy, very efficient when<br>
%              providing an array of lambdas, false by default)<br>
%           param.normalized : (optional, can be set to true if the x_i's have<br>
%              unit l2-norm, false by default)<br>
%           param.strategy (optional, 3 by default)<br>
%                          0: no heuristics, slow  (only for comparison purposes)<br>
%                          1: adjust the constant L on 5% of the data <br>
%                          2: adjust the constant L on 5% of the data + unstable heuristics (this strategy does not work)<br>
%                          3: adjust the constant L on 5% of the data + stable heuristic (this is by far the best choice)<br>
%           param.numThreads (optional, number of threads)<br>
%           param.verbose (optional)<br>
%           param.seed (optional, choice of the random seed)<br>
%<br>
% Output:  W:  double dense p x nlambda matrix<br>
%          optim: optional, double dense 3 x nlambda matrix.<br>
%              first row: values of the objective functions.<br>
%              third row: computational time <br>
%<br>
% Author: Julien Mairal, 2013</span></td></tr>
</table><p>
The following piece of code illustrates how to use this function.
</p><table class="lstframe c011"><tr><td class="mouselstlisting"><span class="c002">clear</span> <span class="c002">all</span>;<br>
n=400000;<br>
p=1000;<br>
density=0.01;<br>
<br>
<span class="c001">% generate random data</span><br>
<span class="c002">format</span> compact;<br>
<span class="c002">randn</span>(<span class="c003">'seed'</span>,0);<br>
<span class="c002">rand</span>(<span class="c003">'seed'</span>,0);<br>
X=<span class="c002">sprandn</span>(p,n,density);<br>
mean_nrm=<span class="c002">mean</span>(<span class="c002">sqrt</span>(<span class="c002">sum</span>(X.^2)));<br>
X=X/mean_nrm;<br>
<br>
<span class="c001">% generate some true model</span><br>
z=double(<span class="c002">sign</span>(<span class="c002">full</span>(<span class="c002">sprandn</span>(p,1,0.05))));<br>
y=X<span class="c003">'*z;<br>
<br>
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%<br>
% EXPERIMENT 1: Lasso<br>
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%<br>
fprintf('</span>EXPERIMENT FOR LASSO\n<span class="c003">');<br>
nrm=sqrt(sum(y.^2));<br>
y=y+0.01*nrm*randn(n,1);    % add noise to the model<br>
nrm=sqrt(sum(y.^2));<br>
y=y*(sqrt(n)/nrm);<br>
<br>
% set optimization parameters<br>
clear param;<br>
param.regul='</span>l1<span class="c003">';        % many other regularization functions are available<br>
param.loss='</span>square<span class="c003">';     % only square and log are available<br>
param.numThreads=-1;    % uses all possible cores<br>
param.normalized=false;  % if the columns of X have unit norm, set to true.<br>
param.strategy=1;        % MISO with all heuristics<br>
                         % 0: no heuristics, slow  (only for comparison purposes)<br>
                         % 1: MISO1: adjust the constant L on 5% of the data  (good for non-strongly convex problems)<br>
                         % 2: adjust the constant L on 5% of the data + unstable heuristics (this strategy does not work)<br>
                         % 3: MISO2: adjust the constant L on 5% of the data + stable heuristic good for strongly-convex problems<br>
                         % 4: MISOmu: best for l2 regularization<br>
param.verbose=true;<br>
<br>
% set grid of lambda<br>
max_lambda=max(abs(X*y))/n;<br>
tablambda=max_lambda*(2^(1/4)).^(0:-1:-20);  % order from large to small<br>
tabepochs=[1 2 3 5 10];  % in this script, we compare the results obtained when changing the number of passes on the data.<br>
param.lambda=tablambda;<br>
%% The problem which will be solved is<br>
%%   min_beta  1/(2n) ||y-X'</span> <span class="c002">beta</span>||_2^2 + lambda ||<span class="c002">beta</span>||_1<br>
<span class="c001">% the problems for different lambdas are solve INDEPENDENTLY in parallel</span><br>
<span class="c002">fprintf</span>(<span class="c003">'EXPERIMENT: ALL LAMBDAS WITHOUT WARM RESTART\n'</span>);<br>
param.warm_restart=false;<br>
param.verbose=false;<br>
param.strategy=1;<br>
param.minibatches=<span class="c002">min</span>(n,<span class="c002">ceil</span>(1/density));  <span class="c001">% size of the minibatches, requires to store twice the size of X  with strategy=1</span><br>
obj=[];<br>
spar=[];<br>
<span class="c002">for</span> ii=1:<span class="c002">length</span>(tabepochs)<br>
   param.epochs=tabepochs(ii);   <span class="c001">% one pass over the data</span><br>
   <span class="c002">fprintf</span>(<span class="c003">'EXP WITH %d PASS OVER THE DATA\n'</span>,tabepochs(ii));<br>
   nlambdas=<span class="c002">length</span>(param.lambda);<br>
   Beta0=<span class="c002">zeros</span>(p,nlambdas);<br>
   <span class="c002">tic</span><br>
   [Beta tmp]=mexIncrementalProx(y,X,Beta0,param);<br>
   <span class="c002">toc</span><br>
   <span class="c002">fprintf</span>(<span class="c003">'Objective functions: \n'</span>);<br>
   obj=[obj; tmp(1,:)];<br>
   obj<br>
   <span class="c002">fprintf</span>(<span class="c003">'Sparsity: \n'</span>);<br>
   spar=[spar; <span class="c002">sum</span>(Beta ~= 0)];<br>
   spar<br>
<span class="c002">end<br>
<br>
<span class="c001">% the problems are here solved sequentially with warm restart<br>
% this seems to be the prefered choice.</span><br>
fprintf</span>(<span class="c003">'EXPERIMENT: SEQUENTIAL LAMBDAS WITH WARM RESTART\n'</span>);<br>
<span class="c002">fprintf</span>(<span class="c003">'A SINGLE CORE IS USED\n'</span>);<br>
param.warm_restart=true;<br>
param.num_threads=1;<br>
obj=[];<br>
spar=[];<br>
<span class="c002">for</span> ii=1:<span class="c002">length</span>(tabepochs)<br>
   param.epochs=tabepochs(ii);   <span class="c001">% one pass over the data</span><br>
   <span class="c002">fprintf</span>(<span class="c003">'EXP WITH %d PASS\n'</span>,tabepochs(ii));<br>
   nlambdas=<span class="c002">length</span>(param.lambda);<br>
   Beta0=<span class="c002">zeros</span>(p,nlambdas);<br>
   <span class="c002">tic</span><br>
   [Beta tmp]=mexIncrementalProx(y,X,Beta0,param);<br>
   <span class="c002">toc</span><br>
   <span class="c002">fprintf</span>(<span class="c003">'Objective functions: \n'</span>);<br>
   obj=[obj; tmp(1,:)];<br>
   obj<br>
   <span class="c002">fprintf</span>(<span class="c003">'Sparsity: \n'</span>);<br>
   spar=[spar; <span class="c002">sum</span>(Beta ~= 0)];<br>
   spar<br>
<span class="c002">end<br>
<br>
<span class="c001">%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%<br>
% EXPERIMENT 2: L2 logistic regression <br>
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%</span><br>
fprintf</span>(<span class="c003">'EXPERIMENT FOR LOGISTIC REGRESSION + l2\n'</span>);<br>
y=<span class="c002">sign</span>(y);<br>
param.regul=<span class="c003">'l2'</span>;        <span class="c001">% many other regularization functions are available</span><br>
param.loss=<span class="c003">'logistic'</span>;     <span class="c001">% only square and log are available</span><br>
param.num_threads=-1;    <span class="c001">% uses all possible cores</span><br>
param.strategy=4;<br>
param.minibatches=1; <span class="c001">% with strategy=4, minibatches is better set to 1<br>
%param.strategy=3;        % MISO with all heuristics</span><br>
<span class="c002">fprintf</span>(<span class="c003">'EXPERIMENT: ALL LAMBDAS WITHOUT WARM RESTART\n'</span>);<br>
param.warm_restart=false;<br>
param.verbose=false;<br>
obj=[];<br>
spar=[];<br>
<span class="c002">for</span> ii=1:<span class="c002">length</span>(tabepochs)<br>
   param.epochs=tabepochs(ii);   <span class="c001">% one pass over the data</span><br>
   <span class="c002">fprintf</span>(<span class="c003">'EXP WITH %d PASS\n'</span>,tabepochs(ii));<br>
   nlambdas=<span class="c002">length</span>(param.lambda);<br>
   Beta0=<span class="c002">zeros</span>(p,nlambdas);<br>
   <span class="c002">tic</span><br>
   [Beta tmp]=mexIncrementalProx(y,X,Beta0,param);<br>
   <span class="c002">toc</span><br>
   yR=repmat(y,[1 nlambdas]);<br>
   <span class="c002">fprintf</span>(<span class="c003">'Objective functions: \n'</span>);<br>
   obj=[obj;tmp(1,:)];<br>
   obj<br>
<span class="c002">end<br>
<br>
<br>
fprintf</span>(<span class="c003">'EXPERIMENT FOR LOGISTIC REGRESSION + l2\n'</span>);<br>
y=<span class="c002">sign</span>(y);<br>
param.regul=<span class="c003">'l2'</span>;        <span class="c001">% many other regularization functions are available</span><br>
param.loss=<span class="c003">'logistic'</span>;     <span class="c001">% only square and log are available</span><br>
param.num_threads=-1;    <span class="c001">% uses all possible cores</span><br>
param.strategy=3;         <span class="c001">% MISO2</span><br>
param.minibatches=<span class="c002">min</span>(n,<span class="c002">ceil</span>(1/density));  <span class="c001">% size of the minibatches, requires to store twice the size of X </span><br>
<span class="c002">fprintf</span>(<span class="c003">'EXPERIMENT: ALL LAMBDAS WITHOUT WARM RESTART\n'</span>);<br>
param.warm_restart=false;<br>
param.verbose=false;<br>
obj=[];<br>
spar=[];<br>
<span class="c002">for</span> ii=1:<span class="c002">length</span>(tabepochs)<br>
   param.epochs=tabepochs(ii);   <span class="c001">% one pass over the data</span><br>
   <span class="c002">fprintf</span>(<span class="c003">'EXP WITH %d PASS\n'</span>,tabepochs(ii));<br>
   nlambdas=<span class="c002">length</span>(param.lambda);<br>
   Beta0=<span class="c002">zeros</span>(p,nlambdas);<br>
   <span class="c002">tic</span><br>
   [Beta tmp]=mexIncrementalProx(y,X,Beta0,param);<br>
   <span class="c002">toc</span><br>
   yR=repmat(y,[1 nlambdas]);<br>
   <span class="c002">fprintf</span>(<span class="c003">'Objective functions: \n'</span>);<br>
   obj=[obj;tmp(1,:)];<br>
   obj<br>
<span class="c002">end</span><br>
<br>
<span class="c001">%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%<br>
% EXPERIMENT 3: L1 logistic regression <br>
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%</span><br>
<br>
y=<span class="c002">sign</span>(y);<br>
<span class="c002">fprintf</span>(<span class="c003">'EXPERIMENT FOR LOGISTIC REGRESSION + l1\n'</span>);<br>
param.regul=<span class="c003">'l1'</span>;        <span class="c001">% many other regularization functions are available</span><br>
param.loss=<span class="c003">'logistic'</span>;     <span class="c001">% only square and log are available</span><br>
param.num_threads=-1;    <span class="c001">% uses all possible cores</span><br>
param.strategy=1;<br>
param.minibatches=<span class="c002">min</span>(n,<span class="c002">ceil</span>(1/density));  <span class="c001">% size of the minibatches, requires to store twice the size of X </span><br>
<span class="c002">fprintf</span>(<span class="c003">'EXPERIMENT: ALL LAMBDAS WITHOUT WARM RESTART\n'</span>);<br>
param.warm_restart=false;<br>
param.verbose=false;<br>
param.lambda=tablambda;<br>
obj=[];<br>
spar=[];<br>
<span class="c002">for</span> ii=1:<span class="c002">length</span>(tabepochs)<br>
   param.epochs=tabepochs(ii);   <span class="c001">% one pass over the data</span><br>
   <span class="c002">fprintf</span>(<span class="c003">'EXP WITH %d PASS\n'</span>,tabepochs(ii));<br>
   nlambdas=<span class="c002">length</span>(param.lambda);<br>
   Beta0=<span class="c002">zeros</span>(p,nlambdas);<br>
   <span class="c002">tic</span><br>
   [Beta tmp]=mexIncrementalProx(y,X,Beta0,param);<br>
   <span class="c002">toc</span><br>
   yR=repmat(y,[1 nlambdas]);<br>
   <span class="c002">fprintf</span>(<span class="c003">'Objective functions: \n'</span>);<br>
   obj=[obj;tmp(1,:)];<br>
   obj<br>
<span class="c002">end</span></td></tr>
</table>
<!--TOC subsection id="sec43" Function mexStochasticProx-->
<h3 class="subsection" id="sec43">5.14  Function mexStochasticProx</h3><!--SEC END --><p>
This implements the stochastic proximal gradient solver [<a href="#mairal15">26</a>].
</p><table class="lstframe c011"><tr><td class="mouselstlisting"><span class="c001">% <br>
% Usage:  [W [W2]]=mexStochasticProx(y,X,W0,param);<br>
%<br>
% Name: mexStochasticProx<br>
%<br>
% Description: mexStochasticProx implements a proximal MM stochastic algorithm <br>
% for composite optimization in a large scale setting.<br>
%        X is a design matrix of size p x n<br>
%        y is a vector of size n <br>
% WARNING, X is transposed compared to the functions mexFista*, and y is a vector<br>
%        param.lambda is a vector that contains nlambda different values of the<br>
%        regularization parameter (it can be a scalar, in that case nlambda=1)<br>
%        W0: is a dense matrix of size p x nlambda   It is in fact ineffective<br>
%        in the current release<br>
%        W: is the output, dense matrix of size p x nlambda<br>
%<br>
%         - if param.loss='square' and param.regul corresponds to a regularization<br>
%           function (currently 'l1' or 'l2'), the following problem is solved<br>
%           w = argmin (1/n)sum_{i=1}^n 0.5(y_i- x_i^T w)^2 + lambda psi(w)<br>
%         - if param.loss='logistic' and param.regul corresponds to a regularization<br>
%           function (currently 'l1' or 'l2'), the following problem is solved<br>
%           w = argmin (1/n)sum_{i=1}^n log(1+ exp(-y_ix_i^T w)) + lambda psi(w)<br>
%           Note that here, the y_i's should be -1 or +1 <br>
%          <br>
%         The current release does not handle intercepts<br>
%<br>
% Inputs: y: double dense vector of size n<br>
%         X: dense or sparse matrix of size p x n<br>
%         W0: dense matrix of size p x nlambda<br>
%         param: struct<br>
%           param.loss (choice of loss)<br>
%           param.regul (choice of regularization function)<br>
%           param.lambda : vector of size nlambda<br>
%           param.iters : number of iterations (n corresponds to one pass over the data)<br>
%           param.minibatches: size of the mini-batches: recommended value is 1<br>
%           param.normalized : (optional, can be set to true if the x_i's have<br>
%              unit l2-norm, false by default)<br>
%           param.weighting_mode : (optional, 1 by default),<br>
%                0:  w_t = (t_0+1)/(t+t_0)<br>
%                1:  w_t = ((t_0+1)/(t+t_0))^(0.75)<br>
%                2:  w_t = ((t_0+1)/(t+t_0))^(5)<br>
%           param.averaging_mode: (optional, false by default)<br>
%                0: no averaging<br>
%                1: first averaging mode for W2<br>
%                2: second averaging mode for W2<br>
%                WARNING: averaging can be very slow for sparse solutions<br>
%           param.determineEta (optional, automatically choose the parameters of the<br>
%             learning weights w_t, true by default) <br>
%           param.t0 (optional, set up t0 for weights w_t = ((1+t0)/(t+t0))^(alpha)<br>
%           param.numThreads (optional, number of threads)<br>
%           param.verbose (optional)<br>
%           param.seed (optional, choice of the random seed)<br>
%<br>
% Output:  W:  double dense p x nlambda matrix (contains the solution without averaging)<br>
%          W2:  double dense p x nlambda matrix (contains the solution with averaging)<br>
%<br>
% Author: Julien Mairal, 2013</span></td></tr>
</table><p>
The following piece of code illustrates how to use this function.
</p><table class="lstframe c011"><tr><td class="mouselstlisting"><span class="c002">clear</span> <span class="c002">all</span>;<br>
n=400000;<br>
p=1000;<br>
density=0.01;<br>
<br>
<span class="c001">% generate random data</span><br>
<span class="c002">format</span> compact;<br>
<span class="c002">randn</span>(<span class="c003">'seed'</span>,0);<br>
<span class="c002">rand</span>(<span class="c003">'seed'</span>,0);<br>
X=<span class="c002">sprandn</span>(p,n,density);<br>
nrm=<span class="c002">sqrt</span>(<span class="c002">sum</span>(X.^2));<br>
X=X/<span class="c002">mean</span>(nrm);<br>
<span class="c001">%mean_nrm=mean();<br>
%X=X/mean_nrm;<br>
<br>
% generate some true model</span><br>
z=double(<span class="c002">sign</span>(<span class="c002">full</span>(<span class="c002">sprandn</span>(p,1,0.05))));<br>
y=X<span class="c003">'*z;<br>
<br>
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%<br>
% EXPERIMENT 1: Lasso<br>
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%<br>
fprintf('</span>EXPERIMENT FOR LASSO\n<span class="c003">');<br>
nrm=sqrt(sum(y.^2));<br>
y=y+0.01*nrm*randn(n,1);    % add noise to the model<br>
nrm=sqrt(sum(y.^2));<br>
y=y*(sqrt(n)/nrm);<br>
<br>
clear param;<br>
param.regul='</span>l1<span class="c003">';        % many other regularization functions are available<br>
param.loss='</span>square<span class="c003">';     % only square and log are available<br>
param.numThreads=-1;    % uses all possible cores<br>
param.normalized=false;  % if the columns of X have unit norm, set to true.<br>
param.averaging_mode=0;  % no averaging, averaging was not really useful in experiments<br>
param.weighting_mode=0;  % weights are in O(1/sqrt(n))  (see help mexStochasticProx)<br>
param.optimized_solver=true;  % best is not to touch this option<br>
param.verbose=false;<br>
<br>
% set grid of lambda<br>
max_lambda=max(abs(X*y))/n;<br>
tablambda=max_lambda*(2^(1/4)).^(0:-1:-20);  % order from large to small<br>
param.lambda=tablambda;    % best to order from large to small<br>
tabepochs=[1 2 3 5 10];  % in this script, we compare the results obtained when varying the number of passes over the data.<br>
<br>
%% The problem which will be solved is<br>
%%   min_beta  1/(2n) ||y-X'</span> <span class="c002">beta</span>||_2^2 + lambda ||<span class="c002">beta</span>||_1<br>
<span class="c002">fprintf</span>(<span class="c003">'EXPERIMENT: ALL LAMBDAS IN PARALLEL\n'</span>);<br>
<span class="c001">% we try different experiments when varying the number of epochs.<br>
% the problems for different lambdas are solve INDEPENDENTLY in parallel</span><br>
obj=[];<br>
objav=[];<br>
<span class="c002">for</span> ii=1:<span class="c002">length</span>(tabepochs)<br>
   param.iters=tabepochs(ii)*n;<br>
   <span class="c002">fprintf</span>(<span class="c003">'EXP WITH %d PASS\n'</span>,tabepochs(ii));<br>
   nlambdas=<span class="c002">length</span>(param.lambda);<br>
   Beta0=<span class="c002">zeros</span>(p,nlambdas);<br>
   <span class="c002">tic</span><br>
   [Beta tmp]=mexStochasticProx(y,X,Beta0,param);<br>
   <span class="c002">toc</span><br>
   yR=repmat(y,[1 nlambdas]);<br>
   <span class="c002">fprintf</span>(<span class="c003">'Objective functions: \n'</span>);<br>
   obj=[obj; 0.5*<span class="c002">sum</span>((yR-X<span class="c003">'*Beta).^2)/n+param.lambda.*sum(abs(Beta))];<br>
   obj<br>
   if param.averaging_mode<br>
      objav=[objav; 0.5*sum((yR-X'</span>*tmp).^2)/n+param.lambda.*<span class="c002">sum</span>(<span class="c002">abs</span>(tmp))];<br>
      objav<br>
   <span class="c002">end</span><br>
   <span class="c002">fprintf</span>(<span class="c003">'Sparsity: \n'</span>);<br>
   <span class="c002">sum</span>(Beta ~= 0)<br>
<span class="c002">end<br>
<br>
<span class="c001">%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%<br>
% EXPERIMENT 2: L2 logistic regression <br>
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%</span><br>
fprintf</span>(<span class="c003">'EXPERIMENT FOR LOGISTIC REGRESSION + l2\n'</span>);<br>
y=<span class="c002">sign</span>(y);<br>
param.regul=<span class="c003">'l2'</span>;<br>
param.loss=<span class="c003">'logistic'</span>;<br>
obj=[];<br>
objav=[];<br>
<span class="c002">for</span> ii=1:<span class="c002">length</span>(tabepochs)<br>
   param.iters=tabepochs(ii)*n;   <span class="c001">% one pass over the data</span><br>
   <span class="c002">fprintf</span>(<span class="c003">'EXP WITH %d PASS\n'</span>,tabepochs(ii));<br>
   nlambdas=<span class="c002">length</span>(param.lambda);<br>
   Beta0=<span class="c002">zeros</span>(p,nlambdas);<br>
   <span class="c002">tic</span><br>
   [Beta tmp]=mexStochasticProx(y,X,Beta0,param);<br>
   <span class="c002">toc</span><br>
   yR=repmat(y,[1 nlambdas]);<br>
   <span class="c002">fprintf</span>(<span class="c003">'Objective functions: \n'</span>);<br>
   obj=[obj;<span class="c002">sum</span>(<span class="c002">log</span>(1.0+<span class="c002">exp</span>(-yR .* (X<span class="c003">'*Beta))))/n+0.5*param.lambda.*sum(abs(Beta.^2))];<br>
   obj<br>
   if param.averaging_mode<br>
      objav=[objav; sum(log(1.0+exp(-yR .* (X'</span>*tmp))))/n+0.5*param.lambda.*<span class="c002">sum</span>(<span class="c002">abs</span>(tmp))];<br>
      objav<br>
   <span class="c002">end<br>
end<br>
<br>
<span class="c001">%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%<br>
% EXPERIMENT 3: L1 logistic regression <br>
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%</span><br>
fprintf</span>(<span class="c003">'EXPERIMENT FOR LOGISTIC REGRESSION + l1\n'</span>);<br>
y=<span class="c002">sign</span>(y);<br>
param.regul=<span class="c003">'l1'</span>;        <span class="c001">% many other regularization functions are available</span><br>
param.loss=<span class="c003">'logistic'</span>;     <span class="c001">% only square and log are available</span><br>
obj=[];<br>
objav=[];<br>
<span class="c002">for</span> ii=1:<span class="c002">length</span>(tabepochs)<br>
   param.iters=tabepochs(ii)*n;   <span class="c001">% one pass over the data</span><br>
   <span class="c002">fprintf</span>(<span class="c003">'EXP WITH %d PASS\n'</span>,tabepochs(ii));<br>
   nlambdas=<span class="c002">length</span>(param.lambda);<br>
   Beta0=<span class="c002">zeros</span>(p,nlambdas);<br>
   <span class="c002">tic</span><br>
   [Beta tmp]=mexStochasticProx(y,X,Beta0,param);<br>
   <span class="c002">toc</span><br>
   yR=repmat(y,[1 nlambdas]);<br>
   <span class="c002">fprintf</span>(<span class="c003">'Objective functions: \n'</span>);<br>
   obj=[obj; <span class="c002">sum</span>(<span class="c002">log</span>(1.0+<span class="c002">exp</span>(-yR .* (X<span class="c003">'*Beta))))/n+param.lambda.*sum(abs(Beta))];<br>
   obj<br>
   if param.averaging_mode<br>
      objav=[objav; sum(log(1.0+exp(-yR .* (X'</span>*tmp))))/n+param.lambda.*<span class="c002">sum</span>(<span class="c002">abs</span>(tmp))];<br>
      objav<br>
   <span class="c002">end</span><br>
   <span class="c002">fprintf</span>(<span class="c003">'Sparsity: \n'</span>);<br>
   <span class="c002">sum</span>(Beta ~= 0)<br>
<span class="c002">end</span></td></tr>
</table>
<!--TOC section id="sec44" A few Functions for manipulating images-->
<h2 class="section" id="sec44">6  A few Functions for manipulating images</h2><!--SEC END --><p>
This functions are not well documented yet
</p>
<!--TOC subsection id="sec45" Function mexExtractPatches-->
<h3 class="subsection" id="sec45">6.1  Function mexExtractPatches</h3><!--SEC END --><table class="lstframe c011"><tr><td class="mouselstlisting"><span class="c001">% <br>
% Usage:   X =mexExtractPatches(I,n,step);<br>
%<br>
% Name: mexExtractPatches<br>
%<br>
% Description: Extract n x n patches spaced every ``step'' pixels from <br>
% an image I of size nx x ny x nchannels<br>
%<br>
% Inputs: I:  double nx x ny x nchannels   <br>
%<br>
% Output: X: double dense matrix<br>
%<br>
% Author: Julien Mairal, 2014</span></td></tr>
</table>
<!--TOC subsection id="sec46" Function mexCombinePatches-->
<h3 class="subsection" id="sec46">6.2  Function mexCombinePatches</h3><!--SEC END --><table class="lstframe c011"><tr><td class="mouselstlisting"><span class="c001">% <br>
% Usage:   I =mexCombinePatches(X,I0,n,step,lambda,normalize);<br>
%<br>
% Name: mexCombinePatches<br>
%<br>
% Description: Combine patches extracted with mexExtractPatches into a new image <br>
%              I = lambda I0 + combined(X) if normalize, averaging is<br>
%              performed; otherwise the patches are just summed.<br>
%<br>
% Inputs: I0:  double nx x ny x nchannels   <br>
%<br>
% Output: I: double nx x ny x nchannels <br>
%<br>
% Author: Julien Mairal, 2014</span></td></tr>
</table>
<!--TOC subsection id="sec47" Function mexConvFistaFlat-->
<h3 class="subsection" id="sec47">6.3  Function mexConvFistaFlat</h3><!--SEC END --><table class="lstframe c011"><tr><td class="mouselstlisting"><span class="c001">% <br>
% Usage:   alpha =mexConvFistaFlat(I,D,alpha0,param);<br>
%<br>
% Name: mexConvFistaFlat<br>
%<br>
% Description: performs convolutional sparse encoding of an<br>
%              image I with a local dictionary D, using FISTA<br>
%              and similar options as mexFistaFlat<br>
%<br>
% Inputs: I:  double nx x ny x nchannels   <br>
%         D:  dictionary  <br>
%         alpha0: initial weights  <br>
%<br>
% Output: alpha: output coefficients<br>
%<br>
% Author: Julien Mairal, 2014</span></td></tr>
</table>
<!--TOC section id="sec48" Miscellaneous Functions-->
<h2 class="section" id="sec48">7  Miscellaneous Functions</h2><!--SEC END -->
<!--TOC subsection id="sec49" Function mexConjGrad-->
<h3 class="subsection" id="sec49">7.1  Function mexConjGrad</h3><!--SEC END --><p>Implementation of a conjugate gradient for solving a linear system <span class="c009">Ax</span>=<span class="c009">b</span>
when <span class="c009">A</span> is positive definite. In some cases, it is faster than the Matlab
function <code>pcg</code>, especially when the library uses the Intel Math Kernel Library.
</p><table class="lstframe c011"><tr><td class="mouselstlisting"><span class="c001">% <br>
% Usage:   x =mexConjGrad(A,b,x0,tol,itermax)<br>
%<br>
% Name: mexConjGrad<br>
%<br>
% Description: Conjugate gradient algorithm, sometimes faster than the <br>
%    equivalent Matlab function pcg. In order to solve Ax=b;<br>
%<br>
% Inputs: A:  double square n x n matrix. HAS TO BE POSITIVE DEFINITE<br>
%         b:  double vector of length n.<br>
%         x0: double vector of length n. (optional) initial guess.<br>
%         tol: (optional) tolerance.<br>
%         itermax: (optional) maximum number of iterations.<br>
%<br>
% Output: x: double vector of length n.<br>
%<br>
% Author: Julien Mairal, 2009</span></td></tr>
</table><p>
The following piece of code illustrates how to use this function.
</p><table class="lstframe c011"><tr><td class="mouselstlisting">A=<span class="c002">randn</span>(5000,500);<br>
A=A<span class="c003">'*A;<br>
b=ones(500,1);<br>
x0=b;<br>
tol=1e-4;<br>
itermax=0.5*length(b);<br>
<br>
tic<br>
for ii = 1:20<br>
x1 = mexConjGrad(A,b,x0,tol,itermax);<br>
end<br>
t=toc;<br>
fprintf('</span>mex-file time: <span class="c001">%fs\n',t);</span><br>
<br>
<span class="c002">tic<br>
for</span> ii = 1:20<br>
x2 = pcg(A,b);<br>
<span class="c002">end</span><br>
t=<span class="c002">toc</span>;<br>
<span class="c002">fprintf</span>(<span class="c003">'Matlab time: %fs\n'</span>,t);<br>
<span class="c002">sum</span>((x1(:)-x2(:)).^2)</td></tr>
</table>
<!--TOC subsection id="sec50" Function mexBayer-->
<h3 class="subsection" id="sec50">7.2  Function mexBayer</h3><!--SEC END --><p>Apply a Bayer pattern to an input image
</p><table class="lstframe c011"><tr><td class="mouselstlisting"><span class="c001">% <br>
% Usage:   x =mexConjGrad(A,b,x0,tol,itermax)<br>
%<br>
% Name: mexConjGrad<br>
%<br>
% Description: Conjugate gradient algorithm, sometimes faster than the <br>
%    equivalent Matlab function pcg. In order to solve Ax=b;<br>
%<br>
% Inputs: A:  double square n x n matrix. HAS TO BE POSITIVE DEFINITE<br>
%         b:  double vector of length n.<br>
%         x0: double vector of length n. (optional) initial guess.<br>
%         tol: (optional) tolerance.<br>
%         itermax: (optional) maximum number of iterations.<br>
%<br>
% Output: x: double vector of length n.<br>
%<br>
% Author: Julien Mairal, 2009</span></td></tr>
</table><p>
The following piece of code illustrates how to use this function.
</p><table class="lstframe c011"><tr><td class="mouselstlisting">A=<span class="c002">randn</span>(5000,500);<br>
A=A<span class="c003">'*A;<br>
b=ones(500,1);<br>
x0=b;<br>
tol=1e-4;<br>
itermax=0.5*length(b);<br>
<br>
tic<br>
for ii = 1:20<br>
x1 = mexConjGrad(A,b,x0,tol,itermax);<br>
end<br>
t=toc;<br>
fprintf('</span>mex-file time: <span class="c001">%fs\n',t);</span><br>
<br>
<span class="c002">tic<br>
for</span> ii = 1:20<br>
x2 = pcg(A,b);<br>
<span class="c002">end</span><br>
t=<span class="c002">toc</span>;<br>
<span class="c002">fprintf</span>(<span class="c003">'Matlab time: %fs\n'</span>,t);<br>
<span class="c002">sum</span>((x1(:)-x2(:)).^2)</td></tr>
</table>
<!--TOC subsection id="sec51" Function mexCalcAAt-->
<h3 class="subsection" id="sec51">7.3  Function mexCalcAAt</h3><!--SEC END --><p>For an input sparse matrix <span class="c009">A</span>, this function returns <span class="c009">AA</span><sup><span class="c007">T</span></sup>. For some reasons, when <span class="c009">A</span> has a lot more columns than rows, this function can be much faster than the equivalent matlab command <code>X*A'</code>. 
</p><table class="lstframe c011"><tr><td class="mouselstlisting"><span class="c001">% <br>
% Usage:   AAt =mexCalcAAt(A);<br>
%<br>
% Name: mexCalcAAt<br>
%<br>
% Description: Compute efficiently AAt = A*A', when A is sparse <br>
%   and has a lot more columns than rows. In some cases, it is<br>
%   up to 20 times faster than the equivalent Matlab expression<br>
%   AAt=A*A';<br>
%<br>
% Inputs: A:  double sparse m x n matrix   <br>
%<br>
% Output: AAt: double m x m matrix <br>
%<br>
% Author: Julien Mairal, 2009</span></td></tr>
</table><p>
The following piece of code illustrates how to use this function.
</p><table class="lstframe c011"><tr><td class="mouselstlisting">A=sprand(200,200000,0.05);<br>
<br>
<span class="c002">tic</span><br>
AAt=mexCalcAAt(A);<br>
t=<span class="c002">toc</span>;<br>
<span class="c002">fprintf</span>(<span class="c003">'mex-file time: %fs\n'</span>,t);<br>
<br>
<span class="c002">tic</span><br>
AAt2=A*A<span class="c003">';<br>
t=toc;<br>
fprintf('</span>matlab time: <span class="c001">%fs\n',t);</span><br>
<br>
<span class="c002">sum</span>((AAt(:)-AAt2(:)).^2)</td></tr>
</table>
<!--TOC subsection id="sec52" Function mexCalcXAt-->
<h3 class="subsection" id="sec52">7.4  Function mexCalcXAt</h3><!--SEC END --><p>
For an input sparse matrix <span class="c009">A</span> and a matrix <span class="c009">X</span>, this function returns <span class="c009">XA</span><sup><span class="c007">T</span></sup>. For some reasons, when <span class="c009">A</span> has a lot more columns than rows, this function can be much faster than the equivalent matlab command <code>X*A'</code>. 
</p><table class="lstframe c011"><tr><td class="mouselstlisting"><span class="c001">% <br>
% Usage:   XAt =mexCalcXAt(X,A);<br>
%<br>
% Name: mexCalcXAt<br>
%<br>
% Description: Compute efficiently XAt = X*A', when A is sparse and has a <br>
%   lot more columns than rows. In some cases, it is up to 20 times <br>
%   faster than the equivalent Matlab expression XAt=X*A';<br>
%<br>
% Inputs: X:  double m x n matrix<br>
%         A:  double sparse p x n matrix   <br>
%<br>
% Output: XAt: double m x p matrix <br>
%<br>
% Author: Julien Mairal, 2009</span></td></tr>
</table><p>
The following piece of code illustrates how to use this function.
</p><table class="lstframe c011"><tr><td class="mouselstlisting">X=<span class="c002">randn</span>(64,200000);<br>
A=sprand(200,200000,0.05);<br>
<br>
<span class="c002">tic</span><br>
XAt=mexCalcXAt(X,A);<br>
t=<span class="c002">toc</span>;<br>
<span class="c002">fprintf</span>(<span class="c003">'mex-file time: %fs\n'</span>,t);<br>
<br>
<span class="c002">tic</span><br>
XAt2=X*A<span class="c003">';<br>
t=toc;<br>
fprintf('</span>mex-file time: <span class="c001">%fs\n',t);</span><br>
<br>
<span class="c002">sum</span>((XAt(:)-XAt2(:)).^2)</td></tr>
</table>
<!--TOC subsection id="sec53" Function mexCalcXY-->
<h3 class="subsection" id="sec53">7.5  Function mexCalcXY</h3><!--SEC END --><p>
For two input matrices <span class="c009">X</span> and <span class="c009">Y</span>, this function returns <span class="c009">XY</span>. 
</p><table class="lstframe c011"><tr><td class="mouselstlisting"><span class="c001">% <br>
% Usage:   Z =mexCalcXY(X,Y);<br>
%<br>
% Name: mexCalcXY<br>
%<br>
% Description: Compute Z=XY using the BLAS library used by SPAMS.<br>
%<br>
% Inputs: X:  double m x n matrix<br>
%         Y:  double n x p matrix   <br>
%<br>
% Output: Z: double m x p matrix <br>
%<br>
% Author: Julien Mairal, 2009</span></td></tr>
</table><p>
The following piece of code illustrates how to use this function.
</p><table class="lstframe c011"><tr><td class="mouselstlisting">X=<span class="c002">randn</span>(64,200);<br>
Y=<span class="c002">randn</span>(200,20000);<br>
<br>
<span class="c002">tic</span><br>
XY=mexCalcXY(X,Y);<br>
t=<span class="c002">toc</span>;<br>
<span class="c002">fprintf</span>(<span class="c003">'mex-file time: %fs\n'</span>,t);<br>
<br>
<span class="c002">tic</span><br>
XY2=X*Y;<br>
t=<span class="c002">toc</span>;<br>
<span class="c002">fprintf</span>(<span class="c003">'mex-file time: %fs\n'</span>,t);<br>
<br>
<span class="c002">sum</span>((XY(:)-XY2(:)).^2)</td></tr>
</table>
<!--TOC subsection id="sec54" Function mexCalcXYt-->
<h3 class="subsection" id="sec54">7.6  Function mexCalcXYt</h3><!--SEC END --><p>
For two input matrices <span class="c009">X</span> and <span class="c009">Y</span>, this function returns <span class="c009">XY</span><sup><span class="c007">T</span></sup>.
</p><table class="lstframe c011"><tr><td class="mouselstlisting"><span class="c001">% <br>
% Usage:   Z =mexCalcXYt(X,Y);<br>
%<br>
% Name: mexCalcXYt<br>
%<br>
% Description: Compute Z=XY' using the BLAS library used by SPAMS.<br>
%<br>
% Inputs: X:  double m x n matrix<br>
%         Y:  double p x n matrix   <br>
%<br>
% Output: Z: double m x p matrix <br>
%<br>
% Author: Julien Mairal, 2009</span></td></tr>
</table><p>
The following piece of code illustrates how to use this function.
</p><table class="lstframe c011"><tr><td class="mouselstlisting">X=<span class="c002">randn</span>(64,200);<br>
Y=<span class="c002">randn</span>(200,20000)<span class="c003">';<br>
<br>
tic<br>
XYt=mexCalcXYt(X,Y);<br>
t=toc;<br>
fprintf('</span>mex-file time: <span class="c001">%fs\n',t);</span><br>
<br>
<br>
<span class="c002">tic</span><br>
XYt2=X*Y<span class="c003">';<br>
t=toc;<br>
fprintf('</span>matlab-file time: <span class="c001">%fs\n',t);</span><br>
<br>
<span class="c002">sum</span>((XYt(:)-XYt2(:)).^2)</td></tr>
</table>
<!--TOC subsection id="sec55" Function mexCalcXtY-->
<h3 class="subsection" id="sec55">7.7  Function mexCalcXtY</h3><!--SEC END --><p>
For two input matrices <span class="c009">X</span> and <span class="c009">Y</span>, this function returns <span class="c009">X</span><sup><span class="c007">T</span></sup><span class="c009">Y</span>.
</p><table class="lstframe c011"><tr><td class="mouselstlisting"><span class="c001">% <br>
% Usage:   Z =mexCalcXtY(X,Y);<br>
%<br>
% Name: mexCalcXtY<br>
%<br>
% Description: Compute Z=X'Y using the BLAS library used by SPAMS.<br>
%<br>
% Inputs: X:  double n x m matrix<br>
%         Y:  double n x p matrix   <br>
%<br>
% Output: Z: double m x p matrix <br>
%<br>
% Author: Julien Mairal, 2009</span></td></tr>
</table><p>
The following piece of code illustrates how to use this function.
</p><table class="lstframe c011"><tr><td class="mouselstlisting">X=<span class="c002">randn</span>(64,200)<span class="c003">';<br>
Y=randn(200,20000);<br>
<br>
tic<br>
XtY=mexCalcXtY(X,Y);<br>
t=toc;<br>
fprintf('</span>mex-file time: <span class="c001">%fs\n',t);</span><br>
<br>
<span class="c002">tic</span><br>
XtY2=X<span class="c003">'*Y;<br>
t=toc;<br>
fprintf('</span>matlab-file time: <span class="c001">%fs\n',t);</span><br>
<br>
<span class="c002">sum</span>((XtY(:)-XtY2(:)).^2)</td></tr>
</table>
<!--TOC subsection id="sec56" Function mexInvSym-->
<h3 class="subsection" id="sec56">7.8  Function mexInvSym</h3><!--SEC END --><p>
For an input symmetric matrices <span class="c009">A</span> in ℝ<sup><span class="c007">n</span> × <span class="c007">n</span></sup>, this function returns <span class="c009">A</span><sup>−1</sup>.
</p><table class="lstframe c011"><tr><td class="mouselstlisting"><span class="c001">% <br>
% Usage:   B =mexInvSym(A);<br>
%<br>
% Name: mexInvSym<br>
%<br>
% Description: returns the inverse of a symmetric matrix A<br>
%<br>
% Inputs: A:  double n x n matrix   <br>
%<br>
% Output: B: double n x n matrix <br>
%<br>
% Author: Julien Mairal, 2009</span></td></tr>
</table><p>
The following piece of code illustrates how to use this function.
</p><table class="lstframe c011"><tr><td class="mouselstlisting">A=<span class="c002">rand</span>(1000,1000);<br>
A=A<span class="c003">'*A;<br>
<br>
tic<br>
B=mexInvSym(A);<br>
t=toc;<br>
fprintf('</span>mex-file time: <span class="c001">%fs\n',t);</span><br>
<br>
<span class="c002">tic</span><br>
B2=<span class="c002">inv</span>(A);<br>
t=<span class="c002">toc</span>;<br>
<span class="c002">fprintf</span>(<span class="c003">'matlab-file time: %fs\n'</span>,t);<br>
<br>
<span class="c002">sum</span>((B(:)-B2(:)).^2)</td></tr>
</table>
<!--TOC subsection id="sec57" Function mexNormalize-->
<h3 class="subsection" id="sec57">7.9  Function mexNormalize</h3><!--SEC END --><table class="lstframe c011"><tr><td class="mouselstlisting"><span class="c001">% <br>
% Usage:   Y =mexNormalize(X);<br>
%<br>
% Name: mexNormalize<br>
%<br>
% Description: rescale the columns of X so that they have<br>
%        unit l2-norm.<br>
%<br>
% Inputs: X:  double m x n matrix   <br>
%<br>
% Output: Y: double m x n matrix <br>
%<br>
% Author: Julien Mairal, 2010</span></td></tr>
</table><p>
The following piece of code illustrates how to use this function.
</p><table class="lstframe c011"><tr><td class="mouselstlisting">X=<span class="c002">rand</span>(100,1000);<br>
<br>
<span class="c002">tic</span><br>
Y=mexNormalize(X);<br>
t=<span class="c002">toc</span>;</td></tr>
</table>
<!--TOC subsection id="sec58" Function mexSort-->
<h3 class="subsection" id="sec58">7.10  Function mexSort</h3><!--SEC END --><table class="lstframe c011"><tr><td class="mouselstlisting"><span class="c001">% <br>
% Usage:   Y =mexSort(X);<br>
%<br>
% Name: mexSort<br>
%<br>
% Description: sort the elements of X using quicksort<br>
%<br>
% Inputs: X:  double vector of size n<br>
%<br>
% Output: Y: double  vector of size n<br>
%<br>
% Author: Julien Mairal, 2010</span></td></tr>
</table><p>
The following piece of code illustrates how to use this function.
</p><table class="lstframe c011"><tr><td class="mouselstlisting">X=<span class="c002">rand</span>(1,300000);<br>
<br>
<span class="c002">tic</span><br>
Y=mexSort(X);<br>
t=<span class="c002">toc</span>;<br>
<span class="c002">toc<br>
<br>
tic</span><br>
Y2=<span class="c002">sort</span>(X,<span class="c003">'ascend'</span>);<br>
t=<span class="c002">toc</span>;<br>
<span class="c002">toc<br>
<br>
sum</span>((Y2(:)-Y(:)).^2)</td></tr>
</table>
<!--TOC subsection id="sec59" Function mexDisplayPatches-->
<h3 class="subsection" id="sec59">7.11  Function mexDisplayPatches</h3><!--SEC END --><p>
Print to the screen a matrix containing as columns image patches.</p>
<!--TOC subsection id="sec60" Function mexCountPathsDAG-->
<h3 class="subsection" id="sec60">7.12  Function mexCountPathsDAG</h3><!--SEC END --><p>
This function counts the number of paths in a DAG using dynamic programming.
</p><table class="lstframe c011"><tr><td class="mouselstlisting"><span class="c001">% <br>
% Usage:   num=mexCountPathsDAG(G);<br>
%<br>
% Name: mexCountPathsDAG<br>
%<br>
% Description: mexCountPathsDAG counts the number of paths <br>
%       in a DAG.<br>
%<br>
%       for a graph G with |V| nodes and |E| arcs,<br>
%       G is a double sparse adjacency matrix of size |V|x|V|,<br>
%       with |E| non-zero entries.<br>
%       (see example in test_CountPathsDAG.m<br>
%<br>
%<br>
% Inputs: G:  double sparse |V| x |V| matrix (full graph)<br>
%<br>
% Output: num: number of paths<br>
%<br>
% Author: Julien Mairal, 2012</span></td></tr>
</table><p>
The following piece of code illustrates how to use this function.
</p><table class="lstframe c011"><tr><td class="mouselstlisting"><span class="c001">% graph corresponding to figure 1 in arXiv:1204.4539v1</span><br>
<span class="c002">fprintf</span>(<span class="c003">'test mexCountPathsDAG\n'</span>);<br>
<span class="c001">% this graph is a DAG</span><br>
G=[0 0 0 0 0 0 0 0 0 0 0 0 0;<br>
   1 0 0 0 0 0 0 0 0 0 0 0 0;<br>
   1 1 0 0 0 0 0 0 0 0 0 0 0;<br>
   1 1 0 0 0 0 0 0 0 0 0 0 0;<br>
   0 0 0 1 0 0 0 0 0 0 0 0 0;<br>
   0 1 1 0 1 0 0 0 0 0 0 0 0;<br>
   0 1 0 0 1 0 0 0 0 0 0 0 0;<br>
   0 0 0 0 0 1 1 0 0 0 0 0 0;<br>
   1 0 0 1 0 0 0 0 0 0 0 0 0;<br>
   1 0 0 0 0 0 0 0 1 0 0 0 0;<br>
   0 0 0 0 0 0 0 0 1 1 0 0 0;<br>
   0 0 0 0 0 0 0 0 1 0 1 0 0;<br>
   0 0 0 0 1 0 0 0 1 0 0 1 0];<br>
G=<span class="c002">sparse</span>(G);<br>
num=mexCountPathsDAG(G);<br>
<span class="c002">fprintf</span>(<span class="c003">'Num of paths: %d\n'</span>,num);</td></tr>
</table>
<!--TOC subsection id="sec61" Function mexRemoveCyclesGraph-->
<h3 class="subsection" id="sec61">7.13  Function mexRemoveCyclesGraph</h3><!--SEC END --><p>
One heuristic to remove cycles from a graph.
</p><table class="lstframe c011"><tr><td class="mouselstlisting"><span class="c001">% <br>
% Usage:   DAG=mexRemoveCycleGraph(G);<br>
%<br>
% Name: mexRemoveCycleGraph<br>
%<br>
% Description: mexRemoveCycleGraph heuristically removes<br>
%       arcs along cycles in the graph G to obtain a DAG.<br>
%       the arcs of G can have weights. The heuristic will<br>
%       remove in priority arcs with the smallest weights.<br>
%<br>
%       for a graph G with |V| nodes and |E| arcs,<br>
%       G is a double sparse adjacency matrix of size |V|x|V|,<br>
%       with |E| non-zero entries. The non-zero entries correspond<br>
%       to the weights of the arcs.<br>
%<br>
%       DAG is a also double sparse adjacency matrix of size |V|x|V|,<br>
%       but the graph is acyclic.<br>
%<br>
%       Note that another heuristic to obtain a DAG from a general <br>
%       graph consists of ordering the vertices.<br>
%<br>
% Inputs: G:  double sparse |V| x |V| matrix<br>
%<br>
% Output: DAG:  double sparse |V| x |V| matrix<br>
%<br>
% Author: Julien Mairal, 2012</span></td></tr>
</table><p>
The following piece of code illustrates how to use this function.
</p><table class="lstframe c011"><tr><td class="mouselstlisting"><span class="c002">fprintf</span>(<span class="c003">'test mexRemoveCyclesGraph\n'</span>);<br>
<span class="c001">% this graph is not a DAG</span><br>
G=[0   0   0   0   0   0   0   0   0   0   0   0   0;<br>
   1   0   0   0.1 0   0   0   0.1 0   0   0   0   0;<br>
   1   1   0   0   0   0.1 0   0   0   0   0   0   0;<br>
   1   1   0   0   0   0   0   0   0   0   0   0.1 0;<br>
   0   0   0   1   0   0   0   0   0   0   0   0   0;<br>
   0   1   1   0   1   0   0   0   0   0   0   0   0;<br>
   0   1   0   0   1   0   0   0   0   0   0   0   0;<br>
   0   0   0   0   0   1   1   0   0   0   0   0   0;<br>
   1   0   0   1   0   0   0   0   0   0   0   0   0;<br>
   1   0   0   0   0   0   0   0   1   0   0   0   0;<br>
   0   0   0   0   0   0   0   0   1   1   0   0   0;<br>
   0   0   0   0   0   0   0   0   1   0   1   0   0;<br>
   0   0   0   0   1   0   0   0   1   0   0   1   0];<br>
G=<span class="c002">sparse</span>(G);<br>
DAG=mexRemoveCyclesGraph(G);<br>
<span class="c002">format</span> compact;<br>
<span class="c002">fprintf</span>(<span class="c003">'Original graph:\n'</span>);<br>
<span class="c002">full</span>(G)<br>
<span class="c002">fprintf</span>(<span class="c003">'New graph:\n'</span>);<br>
<span class="c002">full</span>(DAG)</td></tr>
</table>
<!--TOC subsection id="sec62" Function mexCountConnexComponents-->
<h3 class="subsection" id="sec62">7.14  Function mexCountConnexComponents</h3><!--SEC END --><p>
Count the number of connected components of a subgraph from a graph.
</p><table class="lstframe c011"><tr><td class="mouselstlisting"><span class="c001">% <br>
% Usage:   num=mexCountConnexComponents(G,N);<br>
%<br>
% Name: mexCountConnexComponents<br>
%<br>
% Description: mexCountConnexComponents counts the number of connected<br>
%       components of the subgraph of G corresponding to set of nodes<br>
%       in N. In other words, the subgraph of G by removing from G all<br>
%       the nodes which are not in N.<br>
%<br>
%       for a graph G with |V| nodes and |E| arcs,<br>
%       G is a double sparse adjacency matrix of size |V|x|V|,<br>
%       with |E| non-zero entries.<br>
%       (see example in test_CountConnexComponents.m)<br>
%       N is a dense vector of size |V|. if  N[i] is non-zero,<br>
%       it means that the node i is selected.<br>
%<br>
%<br>
% Inputs: G:  double sparse |V| x |V| matrix (full graph)<br>
%         N:  double full |V| vector.<br>
%<br>
% Output: num: number of connected components.<br>
%<br>
% Author: Julien Mairal, 2012</span></td></tr>
</table><p>
The following piece of code illustrates how to use this function.
</p><table class="lstframe c011"><tr><td class="mouselstlisting"><span class="c001">% graph corresponding to figure 1 in arXiv:1204.4539v1</span><br>
<span class="c002">fprintf</span>(<span class="c003">'test mexCountConnexComponents\n'</span>);<br>
<span class="c001">% this graph is a DAG</span><br>
G=[0 0 0 0 0 0 0 0 0 0 0 0 0;<br>
   1 0 0 0 0 0 0 0 0 0 0 0 0;<br>
   1 1 0 0 0 0 0 0 0 0 0 0 0;<br>
   1 1 0 0 0 0 0 0 0 0 0 0 0;<br>
   0 0 0 1 0 0 0 0 0 0 0 0 0;<br>
   0 1 1 0 1 0 0 0 0 0 0 0 0;<br>
   0 1 0 0 1 0 0 0 0 0 0 0 0;<br>
   0 0 0 0 0 1 1 0 0 0 0 0 0;<br>
   1 0 0 1 0 0 0 0 0 0 0 0 0;<br>
   1 0 0 0 0 0 0 0 1 0 0 0 0;<br>
   0 0 0 0 0 0 0 0 1 1 0 0 0;<br>
   0 0 0 0 0 0 0 0 1 0 1 0 0;<br>
   0 0 0 0 1 0 0 0 1 0 0 1 0];<br>
G=<span class="c002">sparse</span>(G);<br>
nodes=[0 1 1 0 0 1 0 0 1 0 1 1 0];<br>
num=mexCountConnexComponents(G,nodes);<br>
<span class="c002">fprintf</span>(<span class="c003">'Num of connected components: %d\n'</span>,num);<br>
<br>
<span class="c001">% this graph is a not a DAG anymore. This function works<br>
% with general graphs.</span><br>
G=[0 0 0 0 0 0 0 0 0 0 0 0 0;<br>
   1 0 0 0 0 0 0 0 0 0 0 0 0;<br>
   1 1 0 1 0 0 0 0 0 0 0 0 0;<br>
   1 1 0 0 0 0 0 0 0 0 0 0 0;<br>
   0 0 0 1 0 0 0 0 0 0 0 0 0;<br>
   0 1 1 0 1 0 0 0 0 0 0 0 0;<br>
   0 1 0 0 1 0 0 0 0 0 0 0 0;<br>
   0 0 0 0 0 1 1 0 0 0 0 0 0;<br>
   1 0 0 1 0 0 0 0 0 0 0 0 0;<br>
   1 0 0 0 0 0 0 0 1 0 0 0 0;<br>
   0 0 0 0 0 0 0 0 1 1 0 0 0;<br>
   0 0 0 0 0 0 0 0 1 0 1 0 0;<br>
   0 0 0 0 1 0 0 0 1 0 0 1 0];<br>
nodes=[0 1 1 0 0 1 0 0 1 0 1 1 0];<br>
G=<span class="c002">sparse</span>(G);<br>
num=mexCountConnexComponents(G,nodes);<br>
<span class="c002">fprintf</span>(<span class="c003">'Num of connected components: %d\n'</span>,num);<br>
<br>
nodes=[0 1 1 1 0 1 0 0 1 0 1 1 0];<br>
num=mexCountConnexComponents(G,nodes);<br>
<span class="c002">fprintf</span>(<span class="c003">'Num of connected components: %d\n'</span>,num);</td></tr>
</table>
<!--TOC subsection id="sec63" Function mexGraphOfGroupStruct-->
<h3 class="subsection" id="sec63">7.15  Function mexGraphOfGroupStruct</h3><!--SEC END --><table class="lstframe c011"><tr><td class="mouselstlisting"><span class="c001">% <br>
% Usage:   groups =mexGraphOfGroupStruct(gstruct)<br>
%<br>
% Name: mexGraphOfGroupStruct<br>
%<br>
% Description: converts a group structure into the graph structure<br>
%    used by mexProximalGraph, mexFistaGraph or mexStructTrainDL<br>
%<br>
% Inputs: gstruct: the structure of groups as a cell array, one element per node<br>
%     an element is itself a 4 elements cell array:<br>
%       nodeid (&gt;= 0), weight (double), array of vars associated to the node,<br>
%       array of children (nodeis's)<br>
% Output: graph: struct (see documentation of mexProximalGraph)<br>
%<br>
% Author: Jean-Paul CHIEZE, 2012</span></td></tr>
</table>
<!--TOC subsection id="sec64" Function mexGroupStructOfString-->
<h3 class="subsection" id="sec64">7.16  Function mexGroupStructOfString</h3><!--SEC END --><table class="lstframe c011"><tr><td class="mouselstlisting"><span class="c001">% <br>
% Usage:   gstruct =mexGroupStructOfString(s)<br>
%<br>
% Name: mexGroupStructOfString<br>
%<br>
% Description: decode a multi-line string describing "simply" the structure of groups<br>
%    of variables needed by mexProximalGraph, mexProximalTree, mexFistaGraph,<br>
%    mexFistaTree and mexStructTrainDL and builds the corresponding group structure.<br>
%    Each line describes a group of variables as a node of a tree.<br>
%    It has up to 4 fields separated by spaces:<br>
%        node-id node-weight [variables-list] -&gt; children-list<br>
%    Let's define Ng = number of groups, and Nv = number of variables.<br>
%    node-id must be in the range (0 - Ng-1), and there must be Ng nodes<br>
%    weight is a float<br>
%    variables-list : a space separated list of integers, maybe empty,<br>
%         but '[' and '] must be present. Numbers in the range (0 - Nv-1)<br>
%    children-list : a space separated list of node-id's<br>
%        If the list is empty, '-&gt;' may be omitted.<br>
%    The data must obey some rules : <br>
%        - A group contains the variables of the corresponding node and of the whole subtree.<br>
%        - Variables attached to a node are those that are not int the subtree.<br>
%        - If the data destination is a Graph, there may be several independant trees,<br>
%          and a varibale may appear in several trees.<br>
%    If the destination is a Tree, there must be only one tree, the root node<br>
%    must have id == 0 and each variable must appear only once.<br>
%<br>
% Inputs: s:  the multi-lines string<br>
%<br>
% Output: groups: cell array, one element for each node<br>
%                an element is itsel a 4 elements cell array:<br>
%                 nodeid (int &gt;= 0), weight (double), array of vars of the node,<br>
%                array of children (nodeid's)<br>
%<br>
% Author: Jean-Paul CHIEZE, 2012</span></td></tr>
</table>
<!--TOC subsection id="sec65" Function mexReadGroupStruct-->
<h3 class="subsection" id="sec65">7.17  Function mexReadGroupStruct</h3><!--SEC END --><table class="lstframe c011"><tr><td class="mouselstlisting"><span class="c001">% <br>
% Usage:   gstruct =mexReadGroupStruct(file)<br>
%<br>
% Name: mexReadGroupStruct<br>
%<br>
% Description: reads a text file describing "simply" the structure of groups<br>
%    of variables needed by mexProximalGraph, mexProximalTree, mexFistaGraph,<br>
%    mexFistaTree and mexStructTrainDL and builds the corresponding group structure.<br>
%    weight is a float<br>
%    variables-list : a space separated list of integers, maybe empty,<br>
%        but '[' and '] must be present. Numbers in the range (0 - Nv-1)<br>
%    children-list : a space separated list of node-id's<br>
%        If the list is empty, '-&gt;' may be omitted.<br>
%    The data must obey some rules : <br>
%        - A group contains the variables of the corresponding node and of the whole subtree.<br>
%        - Variables attached to a node are those that are not int the subtree.<br>
%        - If the data destination is a Graph, there may be several independant trees,<br>
%           and a varibale may appear in several trees.<br>
%    If the destination is a Tree, there must be only one tree, the root node<br>
%        must have id == 0 and each variable must appear only once.<br>
%<br>
% Inputs: file:  the file name<br>
%<br>
% Output: groups: cell array, one element for each node<br>
%                an element is itsel a 4 elements cell array:<br>
%                nodeid (int &gt;= 0), weight (double), array of vars of the node,<br>
%                array of children (nodeid's)<br>
%<br>
% Author: Jean-Paul CHIEZE, 2012</span></td></tr>
</table>
<!--TOC subsection id="sec66" Function mexTreeOfGroupStruct-->
<h3 class="subsection" id="sec66">7.18  Function mexTreeOfGroupStruct</h3><!--SEC END --><table class="lstframe c011"><tr><td class="mouselstlisting"><span class="c001">% <br>
% Usage:   [permutations groups nbvars] =mexTreeOfGroupStruct(gstruct)<br>
%<br>
% Name: mexTreeOfGroupStruct<br>
%<br>
% Description: converts a group structure into the tree structure<br>
%    used by mexProximalTree, mexFistaTree or mexStructTrainDL<br>
%<br>
% Inputs: gstruct: the structure of groups as a cell array, one element per node<br>
%     an element is itself a 4 lements cell array:<br>
%       nodeid (&gt;= 0), weight (double), array of vars associated to the node,<br>
%       array of children (nodeis's)<br>
% Output: permutations: permutation vector that must be applied to the result of the<br>
%               programm using the tree. Empty if no permutation is needed.<br>
%      tree: struct (see documentation of mexProximalTree)<br>
%      nbvars : number of variables in the tree<br>
%<br>
% Author: Jean-Paul CHIEZE, 2012</span></td></tr>
</table>
<!--TOC subsection id="sec67" Function mexSimpleGroupTree-->
<h3 class="subsection" id="sec67">7.19  Function mexSimpleGroupTree</h3><!--SEC END --><table class="lstframe c011"><tr><td class="mouselstlisting"><span class="c001">% <br>
% Usage:   gstruct =mexSimpleGroupTree(degrees)<br>
%<br>
% Name: mexSimpleGroupTree<br>
%<br>
% Description: makes a structure representing a tree given the<br>
%   degree of each level.<br>
%<br>
% Inputs: degrees:  int vector; degrees(i) is the number of children of each node at level i<br>
%<br>
% Output: group_struct: cell array, one element for each node<br>
%                an element is itsel a 4 elements cell array :<br>
%                 nodeid (int &gt;= 0), weight (double), array of vars attached to the node<br>
%                  (here equal to [nodeid]), array of children (nodeid's)<br>
%<br>
% Author: Jean-Paul CHIEZE, 2012</span></td></tr>
</table>
<!--TOC section id="sec68" Duality Gaps with Fenchel Duality-->
<h2 class="section" id="sec68">A  Duality Gaps with Fenchel Duality</h2><!--SEC END --><p><a id="appendix"></a>
This section is taken from the appendix D of Julien Mairal’s PhD thesis [<a href="#mairal11">19</a>].
We are going to use intensively Fenchel Duality (see [<a href="#borwein">2</a>]).
Let us consider the problem
</p><table class="display dcenter"><tr class="c016"><td class="dcell">
</td><td class="dcell"><table class="display"><tr><td class="dcell c012">&nbsp;</td></tr>
<tr><td class="dcell c012">min</td></tr>
<tr><td class="dcell c012"><span class="c009">w</span> ∈ ℝ<sup><span class="c007">p</span></sup></td></tr>
</table></td><td class="dcell"> [<span class="c007">g</span>(<span class="c009">w</span>) </td><td class="dcell"><table class="display"><tr><td class="dcell c012"><span class="c005">▵</span></td></tr>
<tr><td class="dcell c012">=</td></tr>
<tr><td class="dcell c012">&nbsp;</td></tr>
</table></td><td class="dcell"> <span class="c007">f</span>(<span class="c009">w</span>) + λψ(<span class="c009">w</span>)],<a id="software:eq:prb"></a>
    (47)</td></tr>
</table><p>
We first notice that for all the formulations we have been
interested in, <span class="c007">g</span>(<span class="c009">w</span>) can be rewritten
</p><table class="display dcenter"><tr class="c016"><td class="dcell">
<span class="c007">g</span>(<span class="c009">w</span>) = f(<span class="c009">X</span><sup>⊤</sup><span class="c009">w</span>) + λψ(<span class="c009">w</span>), <a id="software:eq:prb2"></a>
    (48)</td></tr>
</table><p>
where <span class="c009">X</span>=[<span class="c009">x</span><sup>1</sup>,…,<span class="c009">x</span><sup><span class="c007">n</span></sup>] are training vectors, and f is an
appropriated smooth real-valued function of ℝ<sup><span class="c007">n</span></sup>,
and ψ one of the regularization functions we have introduced.</p><p>Given a primal variable <span class="c009">w</span> in ℝ<sup><span class="c007">p</span></sup> and a dual variable κ in
ℝ<sup><span class="c007">n</span></sup>, we obtain using classical Fenchel duality rules [<a href="#borwein">2</a>], 
that the following quantity is a duality gap for problem (<a href="#software%3Aeq%3Aprb">47</a>):
</p><table class="display dcenter"><tr class="c016"><td class="dcell">                δ(<span class="c009">w</span>,κ) </td><td class="dcell"><table class="display"><tr><td class="dcell c012"><span class="c005">▵</span></td></tr>
<tr><td class="dcell c012">=</td></tr>
<tr><td class="dcell c012">&nbsp;</td></tr>
</table></td><td class="dcell"> <span class="c007">g</span>(<span class="c009">w</span>) + f<sup>∗</sup>(κ) + λψ<sup>∗</sup>(−<span class="c009">X</span>κ / λ),
</td></tr>
</table><p>
where f<sup>∗</sup> and ψ<sup>∗</sup> are respectively the Fenchel conjugates
of f and ψ. Denoting by <span class="c009">w</span><sup>⋆</sup> the solution of
Eq. (<a href="#software%3Aeq%3Aprb">47</a>), the duality gap is interesting in the sense that
it upperbounds the difference with the optimal value of the function:
</p><table class="display dcenter"><tr class="c016"><td class="dcell">                δ(<span class="c009">w</span>,κ) ≥  <span class="c007">g</span>(<span class="c009">w</span>)−<span class="c007">g</span>(<span class="c009">w</span><sup>⋆</sup>) ≥ 0.
</td></tr>
</table><p>
Similarly, we will consider pairs of primal-dual variables (<span class="c009">W</span>,<span class="c009">K</span>) when 
dealing with matrices.</p><p>During the optimization, sequences of primal variables <span class="c009">w</span> are available, 
and one wishes to exploit duality gaps for estimating the difference
<span class="c007">g</span>(<span class="c009">w</span>)−<span class="c007">g</span>(<span class="c009">w</span><sup>⋆</sup>). This requires the following components:
</p><ul class="itemize"><li class="li-itemize">
being able to efficiently compute f<sup>∗</sup> and ψ<sup>∗</sup>.
</li><li class="li-itemize">being able to obtain a “good” dual variable κ given a primal
variable <span class="c009">w</span>, such that δ(<span class="c009">w</span>,κ) is close to
<span class="c007">g</span>(<span class="c009">w</span>)−<span class="c007">g</span>(<span class="c009">w</span><sup>⋆</sup>).
</li></ul><p>We suppose that the first point is satisfied (we will detail these computations
for every loss and regularization functions in the sequel), and explain how to
choose κ in general (details will also be given in the sequel).</p><p>Let us first consider the choice that associates with a primal variable <span class="c009">w</span>, the
dual variable 
</p><table class="display dcenter"><tr class="c016"><td class="dcell">
κ(<span class="c009">w</span>) </td><td class="dcell"><table class="display"><tr><td class="dcell c012"><span class="c005">▵</span></td></tr>
<tr><td class="dcell c012">=</td></tr>
<tr><td class="dcell c012">&nbsp;</td></tr>
</table></td><td class="dcell"> ∇ f(<span class="c009">X</span><sup>⊤</sup><span class="c009">w</span>), <a id="software:eq:kappab"></a>
    (49)</td></tr>
</table><p>
and let us compute δ(<span class="c009">w</span>,κ(<span class="c009">w</span>)).
First, easy computations show that for all vectors <span class="c009">z</span> in ℝ<sup><span class="c007">n</span></sup>,
f<sup>∗</sup>(∇f(<span class="c009">z</span>)) = <span class="c009">z</span><sup>⊤</sup>∇f(<span class="c009">z</span>)−f(<span class="c009">z</span>),
which gives
</p><table class="display dcenter"><tr class="c016"><td class="dcell">


     

</td><td class="dcell"><table class="c000 cellpading0"><tr><td class="c015">                δ(<span class="c009">w</span>,κ(<span class="c009">w</span>))</td><td class="c013">=</td><td class="c014"> f(<span class="c009">X</span><sup>⊤</sup><span class="c009">w</span>) + λ ψ(<span class="c009">w</span>) + f<sup>∗</sup>(∇f(<span class="c009">X</span><sup>⊤</sup><span class="c009">w</span>)) + λψ<sup>∗</sup>(−<span class="c009">X</span>∇f(<span class="c009">X</span><sup>⊤</sup><span class="c009">w</span>)/ λ), </td><td class="c015">    (50)</td></tr>
<tr><td class="c015">&nbsp;</td><td class="c013">=</td><td class="c014"> λ ψ(<span class="c009">w</span>) + <span class="c009">w</span><sup>⊤</sup><span class="c009">X</span>∇f(<span class="c009">X</span><sup>⊤</sup><span class="c009">w</span>) + λψ<sup>∗</sup>(−<span class="c009">X</span>∇f(<span class="c009">X</span><sup>⊤</sup><span class="c009">w</span>)/ λ). 
</td><td class="c015">    (51)</td></tr>
</table></td></tr>
</table><p>
We now use the classical Fenchel-Young inequality (see, Proposition
3.3.4 of [<a href="#borwein">2</a>]) on the function ψ, which gives
</p><table class="display dcenter"><tr class="c016"><td class="dcell">                                         </td><td class="dcell"><table class="c000 cellpading0"><tr><td class="c015">                                         δ(<span class="c009">w</span>,κ(<span class="c009">w</span>))  ≥  <span class="c009">w</span><sup>⊤</sup><span class="c009">X</span>∇f(<span class="c009">X</span><sup>⊤</sup><span class="c009">w</span>) − <span class="c009">w</span><sup>⊤</sup><span class="c009">X</span>∇f(<span class="c009">X</span><sup>⊤</sup><span class="c009">w</span>) = 0,
</td></tr>
</table></td></tr>
</table><p>
with equality if and only if −<span class="c009">X</span>∇f(<span class="c009">X</span><sup>⊤</sup><span class="c009">w</span>) belongs to
∂ ψ(<span class="c009">w</span>). Interestingly, we now that first-order optimality
conditions for Eq. (<a href="#software%3Aeq%3Aprb2">48</a>) gives that
−<span class="c009">X</span>∇f(<span class="c009">X</span><sup>⊤</sup><span class="c009">w</span><sup>⋆</sup>) ∈ ∂ ψ(<span class="c009">w</span><sup>⋆</sup>).
We have now in hand a non-negative function <span class="c009">w</span> ↦ δ(<span class="c009">w</span>,κ(<span class="c009">w</span>)) of <span class="c009">w</span>, that
upperbounds <span class="c007">g</span>(<span class="c009">w</span>)−<span class="c007">g</span>(<span class="c009">w</span><sup>⋆</sup>) and satisfying δ(<span class="c009">w</span><sup>⋆</sup>,κ(<span class="c009">w</span><sup>⋆</sup>))=0.</p><p>This is however not a sufficient property to make it a good measure
of the quality of the optimization, and further work is required, 
that will be dependent on f and ψ.
We have indeed proven that δ(<span class="c009">w</span><sup>⋆</sup>,κ(<span class="c009">w</span><sup>⋆</sup>)) is always 0. However, 
for <span class="c009">w</span> different than <span class="c009">w</span><sup>⋆</sup>, δ(<span class="c009">w</span><sup>⋆</sup>,κ(<span class="c009">w</span><sup>⋆</sup>)) can be infinite, 
making it a non-informative duality-gap. The reasons for this can be one of the following:
</p><ul class="itemize"><li class="li-itemize">
The term ψ<sup>∗</sup>(−<span class="c009">X</span>∇f(<span class="c009">X</span><sup>⊤</sup><span class="c009">w</span>)/ λ) might have an infinite value.
</li><li class="li-itemize">Intercepts make the problem more complicated. One can write the formulation with an intercept by
adding a row to <span class="c009">X</span> filled with the value 1, add one dimension to the vector <span class="c009">w</span>, and consider
a regularization function ψ that does regularize the last entry of
<span class="c009">w</span>. This further complexifies the computation of ψ<sup>∗</sup> and its
definition, as shown in the next section.
</li></ul><p>Let us now detail how we proceed to solve these problems, but first without considering the intercept.
The analysis is similar when working with matrices <span class="c009">W</span> instead of vectors <span class="c009">w</span>.
</p>
<!--TOC subsubsection id="sec69" Duality Gaps without Intercepts-->
<h4 class="subsubsection" id="sec69">A.0.1  Duality Gaps without Intercepts</h4><!--SEC END --><p>
Let us show how to compute the Fenchel conjugate of the functions we have introduced.
We now present the Fenchel conjugate of the loss functions f.
</p><ul class="itemize"><li class="li-itemize">
<span class="c010">The square loss</span> 
<table class="display dcenter"><tr class="c016"><td class="dcell"><table class="c000 cellpading0"><tr><td class="c014"><table class="display"><tr class="c016"><td class="dcell">                                         f(<span class="c009">z</span>)=</td><td class="dcell"><table class="display"><tr><td class="dcell c012">1</td></tr>
<tr><td class="hbar"></td></tr>
<tr><td class="dcell c012">2<span class="c007">n</span></td></tr>
</table></td><td class="dcell">||<span class="c009">y</span>−<span class="c009">z</span>||<sub>2</sub><sup>2</sup>, </td></tr>
</table></td></tr>
<tr><td class="c014"><table class="display"><tr class="c016"><td class="dcell">                                            f<sup>∗</sup>(κ)=</td><td class="dcell"><table class="display"><tr><td class="dcell c012"><span class="c007">n</span></td></tr>
<tr><td class="hbar"></td></tr>
<tr><td class="dcell c012">2</td></tr>
</table></td><td class="dcell">||κ||<sub>2</sub><sup>2</sup> + κ<sup>⊤</sup><span class="c009">y</span>.
</td></tr>
</table></td></tr>
</table></td></tr>
</table>
</li><li class="li-itemize"><span class="c010">The logistic loss</span> 
<table class="display dcenter"><tr class="c016"><td class="dcell"><table class="c000 cellpading0"><tr><td class="c014"><table class="display"><tr class="c016"><td class="dcell">                                            f(<span class="c009">z</span>)=</td><td class="dcell"><table class="display"><tr><td class="dcell c012">1</td></tr>
<tr><td class="hbar"></td></tr>
<tr><td class="dcell c012"><span class="c007">n</span></td></tr>
</table></td><td class="dcell"><table class="display"><tr><td class="dcell c012"><span class="c007">n</span></td></tr>
<tr><td class="dcell c012"><span class="c006">∑</span></td></tr>
<tr><td class="dcell c012"><span class="c007">i</span>=1</td></tr>
</table></td><td class="dcell"> log(1+<span class="c007">e</span><sup>−<span class="c007">y</span><sub><span class="c007">i</span></sub><span class="c009">z</span><sub><span class="c007">i</span></sub></sup>) </td></tr>
</table></td></tr>
<tr><td class="c014"><table class="display"><tr class="c016"><td class="dcell">                                               f<sup>∗</sup>(κ)=</td><td class="dcell"><table class="display"><tr class="c016"><td class="dcell">⎧<br>
⎪<br>
⎪<br>
⎨<br>
⎪<br>
⎪<br>
⎩</td><td class="dcell"><table class="c000 cellpading0"><tr><td class="c014">                                               +∞  if  ∃ <span class="c007">i</span>∈ [ 1;<span class="c007">n</span> ]   s.t.   <span class="c007">y</span><sub><span class="c007">i</span></sub>κ<sub><span class="c007">i</span></sub> ∉ [−1,0],</td></tr>
<tr><td class="c014"><table class="display"><tr class="c016"><td class="dcell">                                                  </td><td class="dcell"><table class="display"><tr><td class="dcell c012"><span class="c007">n</span></td></tr>
<tr><td class="dcell c012"><span class="c006">∑</span></td></tr>
<tr><td class="dcell c012"><span class="c007">i</span>=1</td></tr>
</table></td><td class="dcell">(1+<span class="c007">y</span><sub><span class="c007">i</span></sub>κ<sub><span class="c007">i</span></sub>)log(1+<span class="c007">y</span><sub><span class="c007">i</span></sub>κ<sub><span class="c007">i</span></sub>)−<span class="c007">y</span><sub><span class="c007">i</span></sub>κ<sub><span class="c007">i</span></sub>log(−<span class="c007">y</span><sub><span class="c007">i</span></sub>κ<sub><span class="c007">i</span></sub>)  otherwise.
</td></tr>
</table></td></tr>
</table></td></tr>
</table></td></tr>
</table></td></tr>
</table></td></tr>
</table>
</li><li class="li-itemize"><span class="c010">The multiclass logistic loss (or softmax)</span>. The primal variable is now a matrix <span class="c009">Z</span>, in ℝ<sup><span class="c007">n</span> × <span class="c007">r</span></sup>, which represents the product <span class="c009">X</span><sup>⊤</sup><span class="c009">W</span>. We denote by <span class="c009">K</span> the dual variable in ℝ<sup><span class="c007">n</span> × <span class="c007">r</span></sup>.
<table class="display dcenter"><tr class="c016"><td class="dcell"><table class="c000 cellpading0"><tr><td class="c014"><table class="display"><tr class="c016"><td class="dcell">                                                  f(<span class="c009">Z</span>)=</td><td class="dcell"><table class="display"><tr><td class="dcell c012">1</td></tr>
<tr><td class="hbar"></td></tr>
<tr><td class="dcell c012"><span class="c007">n</span></td></tr>
</table></td><td class="dcell"><table class="display"><tr><td class="dcell c012"><span class="c007">n</span></td></tr>
<tr><td class="dcell c012"><span class="c006">∑</span></td></tr>
<tr><td class="dcell c012"><span class="c007">i</span>=1</td></tr>
</table></td><td class="dcell"> log</td><td class="dcell">⎛<br>
⎜<br>
⎝</td><td class="dcell"><table class="display"><tr><td class="dcell c012"><span class="c007">r</span></td></tr>
<tr><td class="dcell c012"><span class="c006">∑</span></td></tr>
<tr><td class="dcell c012"><span class="c007">j</span>=1</td></tr>
</table></td><td class="dcell"> <span class="c007">e</span><sup> <span class="c009">Z</span><sub><span class="c007">ij</span></sub> − <span class="c009">Z</span><sub><span class="c007">i<span class="c010">y</span></span><sub><span class="c007">i</span></sub></sub></sup></td><td class="dcell">⎞<br>
⎟<br>
⎠</td></tr>
</table></td></tr>
<tr><td class="c014"><table class="display"><tr class="c016"><td class="dcell">                                                     f<sup>∗</sup>(<span class="c009">K</span>)=</td><td class="dcell"><table class="display"><tr class="c016"><td class="dcell">⎧<br>
⎪<br>
⎪<br>
⎨<br>
⎪<br>
⎪<br>
⎩</td><td class="dcell"><table class="c000 cellpading0"><tr><td class="c014">                                                     +∞  if  ∃ <span class="c007">i</span> ∈[ 1;<span class="c007">n</span> ]   s.t.   { <span class="c009">K</span><sub><span class="c007">ij</span></sub> &lt; 0  and  <span class="c007">j</span> ≠ <span class="c009">y</span><sub><span class="c007">i</span></sub> }  or  <span class="c009">K</span><sub><span class="c007">i<span class="c010">y</span></span><sub><span class="c007">i</span></sub></sub> &lt; −1,</td></tr>
<tr><td class="c014"><table class="display"><tr class="c016"><td class="dcell">                                                        </td><td class="dcell"><table class="display"><tr><td class="dcell c012"><span class="c007">n</span></td></tr>
<tr><td class="dcell c012"><span class="c006">∑</span></td></tr>
<tr><td class="dcell c012"><span class="c007">i</span>=1</td></tr>
</table></td><td class="dcell"> </td><td class="dcell">⎡<br>
⎢<br>
⎣</td><td class="dcell"><table class="display"><tr><td class="dcell c012">&nbsp;</td></tr>
<tr><td class="dcell c012"><span class="c006">∑</span></td></tr>
<tr><td class="dcell c012"><span class="c007">j</span> ≠ <span class="c009">y</span><sub><span class="c007">i</span></sub></td></tr>
</table></td><td class="dcell"><span class="c009">K</span><sub><span class="c007">ij</span></sub>log(<span class="c009">K</span><sub><span class="c007">ij</span></sub>) + (1+<span class="c009">K</span><sub><span class="c007">i</span> <span class="c009">y</span><sub><span class="c007">i</span></sub></sub>)log(1+<span class="c009">K</span><sub><span class="c007">i</span> <span class="c009">y</span><sub><span class="c007">i</span></sub></sub>)</td><td class="dcell">⎤<br>
⎥<br>
⎦</td><td class="dcell">.
</td></tr>
</table></td></tr>
</table></td></tr>
</table></td></tr>
</table></td></tr>
</table></td></tr>
</table>
</li></ul><p>Our first remark is that the choice Eq. (<a href="#software%3Aeq%3Akappab">49</a>) ensures
that f(κ) is not infinite.</p><p>As for the regularization function, except for the Tikhonov regularization
which is self-conjugate (it is equal to its Fenchel conjugate), we have
considered functions that are norms. There exists therefore a norm ||.|| such
that ψ(<span class="c009">w</span>)=||<span class="c009">w</span>||, and we denote by ||.||<sub>∗</sub> its dual-norm. In such a
case, the Fenchel conjugate of ψ for a vector γ in ℝ<sup><span class="c007">p</span></sup> takes the
form
</p><table class="display dcenter"><tr class="c016"><td class="dcell">                                                        ψ<sup>∗</sup>(γ) = </td><td class="dcell"><table class="display"><tr class="c016"><td class="dcell">⎧<br>
⎨<br>
⎩</td><td class="dcell"><table class="c000 cellpading0"><tr><td class="c014">                                                        0</td><td class="c014"> if  ||γ||<sub>∗</sub>≤ 1, </td></tr>
<tr><td class="c014">                                                           +∞</td><td class="c014"> otherwise.
</td></tr>
</table></td></tr>
</table></td></tr>
</table><p>
It turns out that for almost all the norms we have presented, there exists (i)
either a closed form for the dual-norm or (ii) there exists an 
efficient algorithm evaluating it. The only one which does not conform to this
statement is the tree-structured sum of ℓ<sub>2</sub>-norms, for which we do not know
how to evaluate it efficiently.</p><p>One can now slightly modify the definition of κ
to ensure that ψ<sup>∗</sup>(−<span class="c009">X</span>κ/λ) ≠
+∞. A natural choice is
</p><table class="display dcenter"><tr class="c016"><td class="dcell">   κ(<span class="c009">w</span>) </td><td class="dcell"><table class="display"><tr><td class="dcell c012"><span class="c005">▵</span></td></tr>
<tr><td class="dcell c012">=</td></tr>
<tr><td class="dcell c012">&nbsp;</td></tr>
</table></td><td class="dcell">
min</td><td class="dcell">⎛<br>
⎜<br>
⎝</td><td class="dcell">1,</td><td class="dcell"><table class="display"><tr><td class="dcell c012">λ</td></tr>
<tr><td class="hbar"></td></tr>
<tr><td class="dcell c012">||<span class="c009">X</span>∇f(<span class="c009">X</span><sup>⊤</sup><span class="c009">w</span>)||<sub>∗</sub></td></tr>
</table></td><td class="dcell">⎞<br>
⎟<br>
⎠</td><td class="dcell">∇
f(<span class="c009">X</span><sup>⊤</sup><span class="c009">w</span>), 
</td></tr>
</table><p>
which is the one we have implemented. With this new choice, it is easy to see
that for all vectors <span class="c009">w</span> in ℝ<sup><span class="c007">p</span></sup>, we still have f<sup>∗</sup>(κ) ≠ + ∞, and
finally, we also have δ(<span class="c009">w</span>,κ(<span class="c009">w</span>)) &lt; + ∞ and
δ(<span class="c009">w</span><sup>⋆</sup>,κ(<span class="c009">w</span><sup>⋆</sup>))=0, making it potentially a good
duality gap.</p>
<!--TOC subsubsection id="sec70" Duality Gaps with Intercepts-->
<h4 class="subsubsection" id="sec70">A.0.2  Duality Gaps with Intercepts</h4><!--SEC END --><p>
Even though adding an intercept does seem a simple modification to the original
problem, it induces difficulties for finding good dual variables.</p><p>We recall that having an intercept is equivalent to having a problem of the
type (<a href="#software%3Aeq%3Aprb2">48</a>), by adding a row to <span class="c009">X</span> filled with the value
1, adding one dimension to the vector <span class="c009">w</span> (or one row for matrices <span class="c009">W</span>),
and by using a regularization function that does not depend on the last entry
of <span class="c009">w</span> (or the last row of <span class="c009">W</span>).</p><p>Suppose that we are considering a problem of
type (<a href="#software%3Aeq%3Aprb2">48</a>) of dimension <span class="c007">p</span>+1, but we are using a
regularization function ψ: ℝ<sup><span class="c007">p</span>+1</sup> → ℝ, such that for a
vector <span class="c009">w</span> in ℝ<sup><span class="c007">p</span>+1</sup>, ψ(<span class="c009">w</span>) =<sup><span class="c005">▵</span></sup> ψ(<span class="c009">w</span><sub>[ 1;<span class="c007">p</span> ]</sub>),
where ψ: ℝ<sup><span class="c007">p</span></sup> → ℝ is one of the regularization function we have
introduced. Then, considering a primal variable <span class="c009">w</span>, a dual variable κ, 
and writing γ=<sup><span class="c005">▵</span></sup>−<span class="c009">X</span>κ/λ, we are interested in computing
</p><table class="display dcenter"><tr class="c016"><td class="dcell">   ψ<sup>∗</sup>(γ) = </td><td class="dcell"><table class="display"><tr class="c016"><td class="dcell">⎧<br>
⎨<br>
⎩</td><td class="dcell"><table class="c000 cellpading0"><tr><td class="c014">   +∞  if  γ<sub><span class="c007">p</span>+1</sub> ≠ 0 </td></tr>
<tr><td class="c014">      ψ<sup>∗</sup>(γ<sub>[ 1;<span class="c007">p</span> ]</sub>)  otherwise,
</td></tr>
</table></td></tr>
</table></td></tr>
</table><p>
which means that in order the duality gap not to be infinite, one needs in addition to ensure
that γ<sub><span class="c007">p</span>+1</sub> be zero. Since the last row of <span class="c009">X</span> is filled with ones, this writes
down ∑<sub><span class="c007">i</span>=1</sub><sup><span class="c007">p</span>+1</sup> κ<sub><span class="c007">i</span></sub>=0.
For the formulation with matrices <span class="c009">W</span> and <span class="c009">K</span>, the constraint is similar but for every
column of <span class="c009">K</span>.</p><p>Let us now detail how we proceed for every loss function to find a “good”
dual variable κ satisfying this additional constraint, given a primal
variable <span class="c009">w</span> in ℝ<sup><span class="c007">p</span>+1</sup>, we first define the auxiliary function
</p><table class="display dcenter"><tr class="c016"><td class="dcell">   κ′(<span class="c009">w</span>) </td><td class="dcell"><table class="display"><tr><td class="dcell c012"><span class="c005">▵</span></td></tr>
<tr><td class="dcell c012">=</td></tr>
<tr><td class="dcell c012">&nbsp;</td></tr>
</table></td><td class="dcell"> ∇f(<span class="c009">X</span><sup>⊤</sup><span class="c009">w</span>),
</td></tr>
</table><p>
(which becomes <span class="c009">K</span>′(<span class="c009">W</span>)=<sup><span class="c005">▵</span></sup> ∇f(<span class="c009">X</span><sup>⊤</sup><span class="c009">W</span>) for matrices),
and then define another auxiliary function κ″(<span class="c009">w</span>) as follows,
to take into account the additional constraint ∑<sub><span class="c007">i</span>=1</sub><sup><span class="c007">p</span>+1</sup> κ<sub><span class="c007">i</span></sub>=0.
</p><ul class="itemize"><li class="li-itemize">
<span class="c010">For the square loss</span>, we define another auxiliary function:
<table class="display dcenter"><tr class="c016"><td class="dcell">   κ″(<span class="c009">w</span>) </td><td class="dcell"><table class="display"><tr><td class="dcell c012"><span class="c005">▵</span></td></tr>
<tr><td class="dcell c012">=</td></tr>
<tr><td class="dcell c012">&nbsp;</td></tr>
</table></td><td class="dcell"> κ′(<span class="c009">w</span>) − </td><td class="dcell"><table class="display"><tr><td class="dcell c012">1</td></tr>
<tr><td class="hbar"></td></tr>
<tr><td class="dcell c012"><span class="c007">n</span></td></tr>
</table></td><td class="dcell"><span class="c010">1</span><sub><span class="c007">p</span>+1</sub><sup>⊤</sup>κ′(<span class="c009">w</span>)<span class="c010">1</span><sub><span class="c007">p</span>+1</sub> 
</td></tr>
</table>
where <span class="c010">1</span><sub><span class="c007">p</span>+1</sub> is a vector of size <span class="c007">p</span>+1 filled with ones. This step,
ensures that ∑<sub><span class="c007">i</span>=1</sub><sup><span class="c007">p</span>+1</sup>κ″(<span class="c009">w</span>)<sub><span class="c007">i</span></sub>= 0.
</li><li class="li-itemize"><span class="c010">For the logistic loss</span>, the situation is slightly more complicated since additional constraints are involved in the definition of f<sup>∗</sup>.
<table class="display dcenter"><tr class="c016"><td class="dcell">   κ″(<span class="c009">w</span>) </td><td class="dcell"><table class="display"><tr><td class="dcell c012"><span class="c005">▵</span></td></tr>
<tr><td class="dcell c012">=</td></tr>
<tr><td class="dcell c012">&nbsp;</td></tr>
</table></td><td class="dcell"> <span class="c007">arg</span> <span class="c007">min</span><sub>κ ∈ ℝ<sup><span class="c007">n</span></sup></sub> ||κ−κ′(<span class="c009">w</span>)||<sub>2</sub><sup>2</sup>   s.t.   </td><td class="dcell"><table class="display"><tr><td class="dcell c012"><span class="c007">n</span></td></tr>
<tr><td class="dcell c012"><span class="c006">∑</span></td></tr>
<tr><td class="dcell c012"><span class="c007">i</span>=1</td></tr>
</table></td><td class="dcell"> κ<sub><span class="c007">i</span></sub>=0  and  ∀ <span class="c007">i</span> ∈ [ 1;<span class="c007">n</span> ], κ<sub><span class="c007">i</span></sub> ∈ [−1,0].
</td></tr>
</table>
This problem can be solved in linear-time [<a href="#brucker">3</a>]
using a similar algorithm as for the projection onto the ℓ<sub>1</sub>-ball,
since it is an instance of a <em>quadratic knapsack problem</em>.
</li><li class="li-itemize"><span class="c010">For the multi-class logistic loss</span>, we proceed in a similar way, for every column <span class="c009">K</span><sup><span class="c007">j</span></sup> of <span class="c009">K</span>, <span class="c007">j</span> ∈ [ 1;<span class="c007">r</span> ]:
<div class="flushleft"><table class="display"><tr><td class="dcell">
<span class="c009">K</span><sup>′′ <span class="c007">j</span></sup>(<span class="c009">w</span>) </td><td class="dcell"><table class="display"><tr><td class="dcell c012"><span class="c005">▵</span></td></tr>
<tr><td class="dcell c012">=</td></tr>
<tr><td class="dcell c012">&nbsp;</td></tr>
</table></td><td class="dcell"> <span class="c007">arg</span> <span class="c007">min</span><sub>κ ∈ ℝ<sup><span class="c007">n</span></sup></sub> ||<span class="c009">K</span><sup>′ <span class="c007">j</span></sup>−κ′(<span class="c009">w</span>)||<sub>2</sub><sup>2</sup>   s.t.   </td><td class="dcell"><table class="display"><tr><td class="dcell c012"><span class="c007">n</span></td></tr>
<tr><td class="dcell c012"><span class="c006">∑</span></td></tr>
<tr><td class="dcell c012"><span class="c007">i</span>=1</td></tr>
</table></td><td class="dcell"> κ<sub><span class="c007">i</span></sub>=0  and  </td></tr>
</table></div><div class="flushright"> ∀ <span class="c007">i</span> ∈ [ 1;<span class="c007">n</span> ], {κ<sub><span class="c007">i</span></sub> ≥ 0  if  <span class="c007">j</span> ≠ <span class="c009">y</span><sub><span class="c007">i</span></sub>}  and  {κ<sub><span class="c007">i</span></sub> ≥ −1  if  <span class="c009">y</span><sub><span class="c007">i</span></sub>=<span class="c007">j</span>}.
</div>
</li></ul><p>
When the function ψ is the Tykhonov regularization function, we end the process by setting κ(<span class="c009">w</span>)=κ″(<span class="c009">w</span>).
When it is a norm, we choose, as before for taking into account the constraint ||<span class="c009">X</span>κ||<sub>∗</sub>≤ λ,
</p><table class="display dcenter"><tr class="c016"><td class="dcell">   κ(<span class="c009">w</span>) </td><td class="dcell"><table class="display"><tr><td class="dcell c012"><span class="c005">▵</span></td></tr>
<tr><td class="dcell c012">=</td></tr>
<tr><td class="dcell c012">&nbsp;</td></tr>
</table></td><td class="dcell">  min</td><td class="dcell">⎛<br>
⎜<br>
⎝</td><td class="dcell">1,</td><td class="dcell"><table class="display"><tr><td class="dcell c012">λ</td></tr>
<tr><td class="hbar"></td></tr>
<tr><td class="dcell c012">||<span class="c009">X</span>κ″(<span class="c009">w</span>)||<sub>∗</sub></td></tr>
</table></td><td class="dcell">⎞<br>
⎟<br>
⎠</td><td class="dcell">κ″(<span class="c009">w</span>),
</td></tr>
</table><p>
with a similar formulation for matrices <span class="c009">W</span> and <span class="c009">K</span>.</p><p>Even though finding dual variables while taking into account the intercept
requires quite a lot of engineering, notably implementing a quadratic knapsack
solver, it can be done efficiently.</p><!--TOC section id="sec71" References-->
<h2 class="section" id="sec71">References</h2><!--SEC END --><dl class="thebibliography"><dt class="dt-thebibliography">
<a id="beck">[1]</a></dt><dd class="dd-thebibliography">
A. Beck and M. Teboulle.
A fast iterative shrinkage-thresholding algorithm for linear inverse
problems.
<em>SIAM Journal on Imaging Sciences</em>, 2(1):183–202, 2009.</dd><dt class="dt-thebibliography"><a id="borwein">[2]</a></dt><dd class="dd-thebibliography">
J. M. Borwein and A. S. Lewis.
<em>Convex analysis and nonlinear optimization: Theory and
examples</em>.
Springer, 2006.</dd><dt class="dt-thebibliography"><a id="brucker">[3]</a></dt><dd class="dd-thebibliography">
P. Brucker.
An O(n) algorithm for quadratic knapsack problems.
3:163–166, 1984.</dd><dt class="dt-thebibliography"><a id="candes4">[4]</a></dt><dd class="dd-thebibliography">
E. J. Candès, M. Wakin, and S. Boyd.
Enhancing sparsity by reweighted l1 minimization.
<em>Journal of Fourier Analysis and Applications</em>, 14:877–905,
2008.</dd><dt class="dt-thebibliography"><a id="cherkassky">[5]</a></dt><dd class="dd-thebibliography">
B. V. Cherkassky and A. V. Goldberg.
On implementing the push-relabel method for the maximum flow problem.
<em>Algorithmica</em>, 19(4):390–410, 1997.</dd><dt class="dt-thebibliography"><a id="cotter">[6]</a></dt><dd class="dd-thebibliography">
S. F. Cotter, J. Adler, B. Rao, and K. Kreutz-Delgado.
Forward sequential algorithms for best basis selection.
In <em>IEEE Proceedings of Vision Image and Signal Processing</em>,
pages 235–244, 1999.</dd><dt class="dt-thebibliography"><a id="Cut94">[7]</a></dt><dd class="dd-thebibliography">
A. Cutler and L. Breiman.
Archetypal analysis.
<em>Technometrics</em>, 36(4):338–347, 1994.</dd><dt class="dt-thebibliography"><a id="duchi">[8]</a></dt><dd class="dd-thebibliography">
J. Duchi, S. Shalev-Shwartz, Y. Singer, and T. Chandra.
Efficient projections onto the ℓ<sub>1</sub>-ball for learning in high
dimensions.
In <em>Proceedings of the International Conference on Machine
Learning (ICML)</em>, 2008.</dd><dt class="dt-thebibliography"><a id="efron">[9]</a></dt><dd class="dd-thebibliography">
B. Efron, T. Hastie, I. Johnstone, and R. Tibshirani.
Least angle regression.
<em>Annals of statistics</em>, 32(2):407–499, 2004.</dd><dt class="dt-thebibliography"><a id="friedman">[10]</a></dt><dd class="dd-thebibliography">
J. Friedman, T. Hastie, H. Hölfling, and R. Tibshirani.
Pathwise coordinate optimization.
<em>Annals of statistics</em>, 1(2):302–332, 2007.</dd><dt class="dt-thebibliography"><a id="Friedman2010">[11]</a></dt><dd class="dd-thebibliography">
J. Friedman, T. Hastie, and R. Tibshirani.
A note on the group lasso and a sparse group lasso.
Technical report, Preprint arXiv:1001.0736, 2010.</dd><dt class="dt-thebibliography"><a id="fu">[12]</a></dt><dd class="dd-thebibliography">
W. J. Fu.
Penalized regressions: The bridge versus the Lasso.
<em>Journal of computational and graphical statistics</em>, 7:397–416,
1998.</dd><dt class="dt-thebibliography"><a id="goldberg">[13]</a></dt><dd class="dd-thebibliography">
A. V. Goldberg and R. E. Tarjan.
A new approach to the maximum flow problem.
In <em>Proc. of ACM Symposium on Theory of Computing</em>, pages
136–146, 1986.</dd><dt class="dt-thebibliography"><a id="hoyer">[14]</a></dt><dd class="dd-thebibliography">
P. O. Hoyer.
Non-negative sparse coding.
In <em>Proc. IEEE Workshop on Neural Networks for Signal
Processing</em>, 2002.</dd><dt class="dt-thebibliography"><a id="jenatton3">[15]</a></dt><dd class="dd-thebibliography">
R. Jenatton, J. Mairal, G. Obozinski, and F. Bach.
Proximal methods for sparse hierarchical dictionary learning.
In <em>Proceedings of the International Conference on Machine
Learning (ICML)</em>, 2010.</dd><dt class="dt-thebibliography"><a id="jenatton4">[16]</a></dt><dd class="dd-thebibliography">
R. Jenatton, J. Mairal, G. Obozinski, and F. Bach.
Proximal methods for hierarchical sparse coding.
<em>Journal of Machine Learning Research</em>, 12:2297–2334, 2011.</dd><dt class="dt-thebibliography"><a id="lee2">[17]</a></dt><dd class="dd-thebibliography">
D. D. Lee and H. S. Seung.
Algorithms for non-negative matrix factorization.
In <em>Advances in Neural Information Processing Systems</em>, 2001.</dd><dt class="dt-thebibliography"><a id="maculan">[18]</a></dt><dd class="dd-thebibliography">
N. Maculan and J. R. G. Galdino de Paula.
A linear-time median-finding algorithm for projecting a vector on the
simplex of Rn.
<em>Operations research letters</em>, 8(4):219–222, 1989.</dd><dt class="dt-thebibliography"><a id="mairal11">[19]</a></dt><dd class="dd-thebibliography">
J. Mairal.
<em>Sparse coding for machine learning, image processing and
computer vision</em>.
PhD thesis, Ecole Normale Supérieure, Cachan, 2010.</dd><dt class="dt-thebibliography"><a id="mairal7">[20]</a></dt><dd class="dd-thebibliography">
J. Mairal, F. Bach, J. Ponce, and G. Sapiro.
Online dictionary learning for sparse coding.
In <em>Proceedings of the International Conference on Machine
Learning (ICML)</em>, 2009.</dd><dt class="dt-thebibliography"><a id="mairal9">[21]</a></dt><dd class="dd-thebibliography">
J. Mairal, F. Bach, J. Ponce, and G. Sapiro.
Online learning for matrix factorization and sparse coding.
<em>Journal of Machine Learning Research</em>, 11:19–60, 2010.</dd><dt class="dt-thebibliography"><a id="mairal10">[22]</a></dt><dd class="dd-thebibliography">
J. Mairal, R. Jenatton, G. Obozinski, and F. Bach.
Network flow algorithms for structured sparsity.
In <em>Advances in Neural Information Processing Systems</em>, 2010.</dd><dt class="dt-thebibliography"><a id="mairal13">[23]</a></dt><dd class="dd-thebibliography">
J. Mairal, R. Jenatton, G. Obozinski, and F. Bach.
Convex and network flow optimization for structured sparsity.
<em>Journal of Machine Learning Research</em>, 12:2649–2689, 2011.</dd><dt class="dt-thebibliography"><a id="mairal14">[24]</a></dt><dd class="dd-thebibliography">
J. Mairal and B. Yu.
Supervised feature selection in graphs with path coding penalties and
network flows.
<em>Journal of Machine Learning Research</em>, 2013.</dd><dt class="dt-thebibliography"><a id="mairal16">[25]</a></dt><dd class="dd-thebibliography">
Julien Mairal.
Optimization with first-order surrogate functions.
In <em>International Conference on Machine Learning (ICML)</em>, 2013.</dd><dt class="dt-thebibliography"><a id="mairal15">[26]</a></dt><dd class="dd-thebibliography">
Julien Mairal.
Stochastic majorization-minimization algorithms for large-scale
optimization.
In <em>Advances in Neural Information Processing Systems (NIPS)</em>,
2013.</dd><dt class="dt-thebibliography"><a id="mallat4">[27]</a></dt><dd class="dd-thebibliography">
S. Mallat and Z. Zhang.
Matching pursuit in a time-frequency dictionary.
<em>IEEE Transactions on Signal Processing</em>, 41(12):3397–3415,
1993.</dd><dt class="dt-thebibliography"><a id="meinshausen">[28]</a></dt><dd class="dd-thebibliography">
N. Meinshausen and P. Buehlmann.
Stability selection.
Technical report.
ArXiv:0809.2932.</dd><dt class="dt-thebibliography"><a id="obozinski">[29]</a></dt><dd class="dd-thebibliography">
G. Obozinski, B. Taskar, and M.I. Jordan.
Joint covariate selection and joint subspace selection for multiple
classification problems.
<em>Statistics and Computing</em>, pages 1–22.</dd><dt class="dt-thebibliography"><a id="osborne">[30]</a></dt><dd class="dd-thebibliography">
M. R. Osborne, B. Presnell, and B. A. Turlach.
On the Lasso and its dual.
<em>Journal of Computational and Graphical Statistics</em>,
9(2):319–37, 2000.</dd><dt class="dt-thebibliography"><a id="sprechmann">[31]</a></dt><dd class="dd-thebibliography">
P. Sprechmann, I. Ramirez, G. Sapiro, and Y. C. Eldar.
Collaborative hierarchical sparse modeling.
Technical report, 2010.
Preprint arXiv:1003.0400v1.</dd><dt class="dt-thebibliography"><a id="tibshirani2">[32]</a></dt><dd class="dd-thebibliography">
R. Tibshirani, M. Saunders, S. Rosset, J. Zhu, and K. Knight.
Sparsity and smoothness via the fused lasso.
<em>Journal of the Royal Statistical Society Series B</em>,
67(1):91–108, 2005.</dd><dt class="dt-thebibliography"><a id="tropp3">[33]</a></dt><dd class="dd-thebibliography">
J. A. Tropp.
Algorithms for simultaneous sparse approximation. part ii: Convex
relaxation.
<em>Signal Processing, special issue "Sparse approximations in
signal and image processing"</em>, 86:589–602, April 2006.</dd><dt class="dt-thebibliography"><a id="tropp2">[34]</a></dt><dd class="dd-thebibliography">
J. A. Tropp, A. C. Gilbert, and M. J. Strauss.
Algorithms for simultaneous sparse approximation. part i: Greedy
pursuit.
<em>Signal Processing, special issue "sparse approximations in
signal and image processing"</em>, 86:572–588, April 2006.</dd><dt class="dt-thebibliography"><a id="weisberg">[35]</a></dt><dd class="dd-thebibliography">
S. Weisberg.
<em>Applied Linear Regression</em>.
Wiley, New York, 1980.</dd><dt class="dt-thebibliography"><a id="wu">[36]</a></dt><dd class="dd-thebibliography">
T. T. Wu and K. Lange.
Coordinate descent algorithms for Lasso penalized regression.
<em>Annals of Applied Statistics</em>, 2(1):224–244, 2008.</dd><dt class="dt-thebibliography"><a id="ChenCVPR">[37]</a></dt><dd class="dd-thebibliography">
J. Mairal Y. Chen and Z. Harchaoui.
Fast and robust archetypal analysis for representation learning.
In <em>Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition (CVPR)</em>, 2014.</dd><dt class="dt-thebibliography"><a id="yuan">[38]</a></dt><dd class="dd-thebibliography">
M. Yuan and Y. Lin.
Model selection and estimation in regression with grouped variables.
<em>Journal of the Royal Statistical Society Series B</em>, 68:49–67,
2006.</dd><dt class="dt-thebibliography"><a id="zou">[39]</a></dt><dd class="dd-thebibliography">
H. Zou and T. Hastie.
Regularization and variable selection via the elastic net.
<em>Journal of the Royal Statistical Society Series B</em>,
67(2):301–320, 2005.</dd></dl><!--CUT END -->
<!--HTMLFOOT-->
<!--ENDHTML-->
<!--FOOTER-->
<hr style="height:2"><blockquote class="quote"><em>This document was translated from L<sup>A</sup>T<sub>E</sub>X by
</em><a href="http://hevea.inria.fr/index.html"><em>H<span class="c005"><sup>E</sup></span>V<span class="c005"><sup>E</sup></span>A</em></a><em>.</em></blockquote></body>
</html>
